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FOREWORD 


The  Battlefield  Information  Systems  Technical  Area  is  concerned  with  the 
demands  of  the  future  battlefields  for  increased  human-machine  capacity  to 
acquire,  transmit,  process,  disseminate,  and  utilize  information.  Research 
is  focused  on  interface  problems  and  interactions  within  command,  control, 
and  intelligence  centers  and  is  concerned  with  such  areas  as  tactical  sym¬ 
bology,  user-oriented  systems,  information  management,  staff  operations  and 
procedures,  and  sensor  systems  integration  and  utilization. 

One  area  of  special  interest  is  the  development  of  procedures  to  sup¬ 
port  and  enhance  the  decision-making  process  within  command,  control,  and 
intelligence  centers.  The  current  effort  summarizes  guidelines  for  the  ef¬ 
fective  design,  implementation,  and  evaluation  of  decision-analytic  aids. 

Also  presented  in  the  framework  from  which  these  guidelines  were  developed. 
This  framework  identifies  three  interfaces  essential  to  the  integration  of 
decision  aids  into  organizations.  This  report,  therefore,  should  assist  de¬ 
velopers  in  integrating  decision  aids  into  user  organizations,  resulting  in 
improved  decision  making. 

Research  in  decision  aiding  is  conducted  as  an  in-hcuse  effort  with  ad¬ 
ditional  support  from  contracting  organizations  that  are  selected  for  their 
unique  contributions  to  this  area.  This  effort  is  responsive  to  the  re¬ 
quirements  of  Army  Project  2Q263739A793  and  was  managed  through  the  Cyber¬ 
netics  Technology  Office  of  DARPA. 


EDGAR  M.  JOHNSON 
Technical  Director 
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ISSUES  IN  THE  DESIGN  AND  EVALUATION  OF  DECISION-ANALYTIC  AIDS 


EXECUTIVE  SUMMARY 


Requirement : 

To  formulate  and  demonstrate  guidelines  for  developers  to  use  in  the  de¬ 
sign,  implementation,  and  evaluation  of  decision  aids. 


Approach : 


The  guidelines  were  developed  from  a  framework  based  on  three 
essential  to  the  integration  of  decision  aids  in  organizations, 
terfaces  refer  to  the  contact  points  between  the  user  and  the  aid, 
and  the  decision-making  organization,  and  the  organization  and  its 
environment. 


interfaces 
These  in- 
the  aid 
associated 


Product : 

Design  issues  arising  from  the  interfaces  are  addressed.  Special  atten¬ 
tion  is  devoted  to  (1)  the  immediate  behavioral  effects  of  decision  aids  and 
the  engineering  of  aid  software  and  hardware  so  as  to  minimize  adverse  conse¬ 
quences  and  (2)  the  importance  of  user  involvement  in  aid  design  to  ensure 
the  understanding  and  commitment  necessary  to  adopt  a  different  decision¬ 
making  approach,  as  well  as  to  tailor  the  aid  to  the  users'  particular  needs. 
Evaluation  issues  are  considered,  with  a  focus  on  four  major  areas:  (1)  fac¬ 
tors  at  each  of  the  three  interfaces  that  make  aids  effective,  (2)  evaluation 
settings  (their  similarity  to  operational  settings,  the  amount  of  control 
they  provide,  and  their  costs),  (3)  methods  for  obtaining  measures  of  effec¬ 
tiveness,  and  (4)  control  conditions  required  for  adequate  evaluation. 


Utilization: 

This  report  should  provide  decision  aid  designers  with  guidelines  and 
issues  of  concern  in  the  cycle  of  development,  from  conceptualization  and 
implementation  to  evaluation  and  revision  of  the  aid.  The  information  should 
assist  developers  in  integrating  decision  aids  into  the  intended  organiza¬ 
tions  and  result  in  improved  organizational  decision  making. 
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ISSUES  IN  THE  DESIGN  AND  EVALUATION  OF  DECISION-ANALYTIC  AIDS 


INTRODUCTION 

During  recent  years,  decision  analysis  has  emerged  as  a  highly  valuabi 
technology  for  allowing  decision  makers  to  formulate  important  problems  ir. 
logical  framework,  incorporating  factual  as  well  as  judgmental  information 
to  arrive  at  consistent,  realistic  solutions.  Computers  have  served  well  a 
aids  to  calculation,  disj lay,  editing,  and  memory  functions.  On  the  basis 
of  previous  success,  organizations  are  beginning  to  develop  com;  ut er-based 
decision-analytic  aids  with  stand-alone  ca[  abilities  for  routine  use  by  in¬ 
ternal  analysts  and  decision  makers  without  outside  consultation.  Although 
some  stand-alone  decision  aids  have  been  quite  successful,  o tilers  have  n  t 
been  utilized  by  their  p  rospective  us  ~ rs .  Hie  purpose  of  this  re: ort  is  t. 
provide  guidelines  for  the  effective  design,  imp  lementation ,  and  evaluation 
of  such  decision  aids. 


Throughout  the  following  sections,  the  term  decision  ir  decision- 

analytic  aid  refers  to  a  computer  that  has  been  t  roqramed  assist  m  form 
iating  and  exercising  decision-theoretic  models.  These  include  different 
types  of  multi-attribute  utility  assessment  models  and  tradi^  ,.ul  decision 
theoretic  tree  models  requiring  probability  and  utility  assessments.  Aids 
may  take  on  a  variety  of  forms,  from  the  simp  lest  of  clerical  devices  imp.  le 
men ted  on  micro-  or  mini-computers  (special -purpose  routines  to  perform  cal 
culations  and  to  display  or  store  results)  to  the  most  sophisticated,  state- 
of-the-art,  larqe-seale  computer  implementations  (general-purpose  aids  that 
help  the  user  structure  a  wide  variety  of  problems  ,  search  through  large 
data  bases,  and  perform  complex  analyses).  But  whatever  their  role,  de¬ 
cision  aids  are  designed  to  provide  one  or  both  of  the  following  primary 
benef its : 


•  improved  decision  quality — the  assurance  that  a  decision  is  logical 
based  on  a  consistent,  explicit,  and  realistic  set  of  assumptions; 
and 

•  lower  decision  costs — a  saving  in  some  critical  resource  (time, 
money,  personnel,  etc.)  ,  compared  with  the  unaided  decision  process 

To  be  sure,  additional  benefits  may  accrue:  The  decision  maker  may  develop 
greater  understanding  of  the  overall  problem  area,  or  may  find  computer- 
aided  solutions  easier  to  implement,  but  unless  a  device  either  improves  or 
facilitates  decision  making,  it  cannot  properly  be  termed  a  decision  aid.' 

Figure  1  is  a  pictorial  representation  of  the  framework  for  considenn 
issues  relevant  to  the  design  and  evaluation  of  decision  aids.  These  issue 
arise  at  three  interfaces  represented  in  the  figure.  The  first  interface  i 
between  the  decision  aid  and  the  user.  Here,  the  issue  is  the  extent  to 
which  characteristics  of  the  aid  facilitate  or  hinder  its  usability.  The 
second  interface  is  between  the  user  (and  decision  aid)  and  the  larger 
decision-making  organization.  Here,  the  question  is  to  what  extent  the  de¬ 
cision  aid  facilitates  the  decision-making  processes  of  the  organization. 
The  third  interface  is  between  the  decision-making  organization  and  the 
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environment.  Here,  the  issue  is  whether  the  aid  1  in:  roves,  the  njh'.y 
organisation's  decision  making.  The  sections  of  this  ret-ort  season  i .. 
address  the  issues,  at  each  of  the  three  interfaces. 


DECISION-MAKING 

ORGANIZATION 


LI. 
— J 


; 

USER 

- X 

DECISION 

i 

AIL 

7 

<> 

1  '  I 

1  EN VI  RONV.ENT  ! 

Kiaurc  1.  framew.  rk  for  consi  do  ring  issue?  relevant  to  the  dor  i 
evaluation  of  decision  aids. 


The  sect  is  n  on  behavioral  engineering  addresses  the  first  ir.tei  fa 
startm a  with  tine  following  :  remise  about  hur.an  behavior:  Lt  \ 

fits,  especially  uncertain  and  unattributable  benefits  ,  are  none ra I ly 
weighted  when  cor,:  a  red  with  immediately  observable  effects.  Thus,  ow 
though  a  decision  aid  my  be  objectively  icknowl  O  d  oj  O  C  JS  vl  W  <...•  2*  tn  V  Li  it 
vestment  of  time  and  effort,  the  user's  immediate  behavic i  may  be  dor; 
L  y  she  rt- range  i  erceptions  of  increased  workload  and  by  feel  inci<=  .  :  i 
ir.natier.ee,  frustration,  or  embarrassment  that  stem  not  from  the  :  : .  so 
self,  but  from  its  implementation .  As  a  result,  specific  efforts  rs.rt 
nude  to  analyze  the  immediate  behavioral  effects  of  decision  aid?  and 
engineer  those  aids  so  as  to  minimize  or  reverse  their  adverse  censcuu 
Some  of  the  major  behavioral  j.  roblems  typical  of  current  decision  aids 
discussed ,  and  possible  solutions  to  these  problems  are  offered. 

The  section  on  involving  users  in  the  develoj  ment  of  decision  aid 
dresses  the  second  interface.  The  thesis  of  this  section  is  that  deci 
aids  will  seldom  achieve  stand-alone  status  unless  eventual  users  (bet 
hands-on  users  and  decision  makers)  are  involved  in  their  development  . 
Solving  behavioral  problems  at  the  first  interface  is  necessary,  but  u 
sufficient,  for  aid  implementation .  User  involvement  in  aid  desieu  is 
essential  for  implementation,  for  this  involvement  develops  the  under? 
inq  and  commitment  necessary  for  implementing  a  different  decisi on-rr.ak 
api_ roach  and  tailors  aid  characteristics  to  the  users'  needs  wit)-, it:  th 
organizational  context.  Two  sources  of  support  for  this  position  are 
sented:  (1)  the  recent  systematic  evaluation  of  a  decision  aid  novel..- 


2 


for  the  Operations  Directorate  of  the  Joint  Chiefs  of  Staff  and  (2)  the  his¬ 
tory  of  model  implementation  in  the  field  of  operations  research  and  manage¬ 
ment  science.  Although  the  need  for  user  involvement  appears  obvious,  these 
sources  indicated  that  user  involvement  is  often  neglected  in  the  development 
of  analytical  decision  aids,  frequently  resulting  in  unsuccessful  implementa¬ 
tion  efforts. 

The  final  section  focuses  not  on  the  design  and  implementation  of  de¬ 
cision  aids,  but  on  decision  aid  evaluation.  In  particular,  this  section 
focuses  on  four  major  aspects  of  decision  aid  evaluation.  The  first  aspect 
concerns  the  factors  that  make  a  decision  aid  effective.  These  factors  de¬ 
termine  which  general  measures  of  effectiveness  are  employed  to  evaluate  the 
aid  in  relation  to  previously  specified  objectives.  The  second  aspect  con¬ 
cerns  the  settings  for  decision  aid  evaluations.  Evaluation  settings  differ 
in  their  similarity  to  the  expected  operational  setting,  the  amount  of  con¬ 
trol  they  provide,  and  their  costs;  such  differences  affect  the  extent  to 
which  certain  measures  of  effectiveness  can  be  collected  and  analyzed  during 
the  evaluation.  The  third  aspect  concerns  the  different  methods  for  obtain¬ 
ing  measures  of  effectiveness.  The  fourth  aspect  addresses  the  problem  of 
developing  adequate  control  (or  contrasting)  conditions  for  the  effective 
evaluation  of  decision  aids.  Each  of  the  four  aspects  addresses  issues  at 
each  of  the  three  interfaces. 

It  is  hoped  that  this  rep'ort  will  provide  effective  guidelines  for  the 
design  and  evaluation  of  decision  aids.  The  authors  realize  that  these 
guidelines  will  not  answer  all  the  questions  of  potential  developers  and 
users  of  decision  aids,  for  the  development  of  such  aids  is  less  than  one 
decade  old.  The  report  does  identify,  however,  those  issues  in  development 
and  evaluation  that  have  arisen  in  the  work  of  decision  analysts  over  the 
past  few  years.  Such  information  should  assist  developers  in  integrating 
decision  aids  into  their  organizations  and,  in  turn,  result  in  improved  or¬ 
ganizational  decision  making. 


BEHAVIORAL  ENGINEERING:  CHALLENGE  TO  DECISION  AID  DESIGN 

The  historical  failure  of  most  decision  aids  to  generate  user  enthusi¬ 
asm  can  be  largely  explained  in  terms  of  a  well-known  principle  of  behavioral 
psychology:  Immediate,  certain,  directly  observable  effects  have  high,  im¬ 

pacts  on  behavior,  whereas  delay,  uncertainty,  and  indirectness  of  results 
can  reduce  the  perceived  impact  of  rewards  (benefits)  and  punishments  (costs) . 
If  the  behavior  in  question  is  decision  aid  usage,  the  costs  consist  of  im¬ 
mediate  expenditures  of  time,  effort,  and  attention  in  a  stressful,  time- 
constrained  situation.  The  benefits,  in  contrast,  are  deferred  until  the 
analysis  is  complete  and  the  decision  implemented;  even  then,  uncertain  fu¬ 
ture  events  may  make  a  rational  decision  look  bad.  Furthermore,  many  de¬ 
cisions  have  their  primary  effect  on  other  members  of  the  organization,  so 
that  only  indirect  feedback  reaches  the  decision  maker.  Thus,  the  immediate 
process-related  cost  factors  dominate  the  behavioral  environment  of  decision 
aiding,  and  the  results-related  benefits  have  less  impact  than  they  should. 

This  effect  is  compounded  by  the  fact  that  a  successful  decision  analy¬ 
sis  is  not  a  single  event,  but  a  complex,  prolonged  sequence  of  behavior 
that  requires  continuously  high  levels  of  motivation  and  attention.  There 
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are  many  opportunities  for  the  user  to  become  bored,  contused,  or  discour¬ 
aged,  and  any  one  of  these  may  cause  the  user  to  terminate  the  analysis  or, 
even  worse,  may  lead  to  a  half-hearted  attempt  at  analysis  that  not  only 
compounds  the  behavioral  problems,  but  also  increases  the  risk  of  unnoticed 
analytic  errors.  It  is  not  sufficient  that  the  long-range  benefits  of  using 
a  decision  aid  outweigh  its  operating  costs;  at  every  point  in  the  j rocess , 
the  user  must  p>erceive  immediate  rewards  to  be  motivated  to  continue  the 
analysis . 

This  principle  is  illustrated  by  the  role  of  a  professional  decision 
analyst  in  a  clinical  decision  analysis  pro;ject.  Even  though  the  client  has 
already  made  a  conscious  decision  to  employ  the  decision  analyst's  methodol¬ 
ogy,  the  analyst  must  be  prepared  to  deal  with  occasional  episodes  of  con¬ 
fusion,  fatigue,  boredom,  impatience,  and  discouragement.  Without  proper 
treatment,  these  problems  may  delay  the  analysis,  degrade  the  quality  of  the 
outcome,  or  reduce  the  client's  confidence  in  the  process.  Therefore,  the 
analyst  must  be  more  than  technically  proficient;  the  analyst  must  be  sensi¬ 
tive  to  the  client's  mental  and  emotional  state  and  must  have  the  flexibil¬ 
ity  to  adapt  the  direction  and  timinq  of  the  analysis  accordingly.  Instead 
of  proceeding  in  a  linear  fashion  through  an  analysis,  the  analyst  may  need 
to  stop  and  review  results,  explain  the  procedures  more  carefully,  repeat  a 
portion  of  the  analysis,  call  for  a  break,  or  simply  administer  some  reas¬ 
surance  and  encouragement  to  continue.  In  short,  client  motivation  and  at¬ 
tention  are  as  basic  to  the  function  of  the  decision  analyst  as  is  technical 
performance,  and  perhaps  even  more  important,  as  it  is  easier  to  recover 
from  a  technical  error  than  from  a  motivational  one. 

It  is  a  major  challenge  for  the  professionally  trained  decision  analyst 
to  maintain  motivation  and  attention  consistently,  but  it  is  an  ever,  greater 
challenge  for  the  decision  aid  to  accomp'lish  the  same  task  because  the  aid 
is  not  able  to  observe  the  user's  emotional  and  mental  state,  cannot  infer 
and  adapt  to  the  user's  unique  personal  characteristics,  and  has  no  personal 
credibility  in  a  leadership  role.  In  order  to  maintain  motivation  and  atten¬ 
tion,  the  computerized  decision  aid  must  compensate  for  these  shortcomings  by 
capitalizing  on  its  strengths--speed ,  precision,  memory  capacity,  and  the 
ability  to  generate  neat  and  effective  output  displays--to  minimize  the  per¬ 
ceived  costs  of  usage,  and  to  offset  those  costs  with  even  greater  immediate 
benefits. 

During  the  past  decade,  two  microprocessor-based  technologies  have  had 
dramatically  different  fates.  Commercially  produced  video  games  (e.g., 
"Por.g,"  "Tank  War,"  and  "Space  Invaders”)  have  achieved  rapid  and  widespread 
acceptance  in  the  entertainment  market,  while  computerized  decision  aids  have 
generated  very  little  enthusiasm  among  their  intended  users.  The  play- 
versus-work  distinction  by  itself  does  not  explain  the  divergent  fates  of 
these  two  young  technologies;  but  in  their  efforts  to  capture  a  highly  com¬ 
petitive  market,  the  designers  of  video  games  have  incorporated  features  of 
behavioral  engineering  that  decision  aid  designers  have  largely  ignored. 

Thus,  the  success  of  the  video  games  can  be  regarded  as  a  challenge  to  the 
decision  aid  designers,  suggesting  new  directions  to  guide  further  develop¬ 
ment,  while  offering  optimistic  evidence  for  eventual  success. 

The  major  contrast  between  the  current  generation  of  decision  aids  and 
the  far  more  successful  video  games  is  that  the  former  depend  in  large  part 
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on  the  user's  ability  to  endure  unpleasant  work  in  order  to  achieve  a  worth¬ 
while  long-range  result,  whereas  the  latter  focus  on  more  immediate  features: 
simplicity  of  operation,  speed  of  response,  direct  sensory  feedback,  and  fre¬ 
quent  rewards.  This  section  suggests  several  ways  in  which  decision  aids 
could  be  desiqned  to  take  advantage  of  the  behavioral  principles  that  favor 
the  lac  tor  approach. 


Training,  Initial  Access,  and  Startup 

Probably  the  most  critical  hurdle  is  to  get  the  user  to  sit  down  in 
front  of  the  machine,  turn  it  on,  and  start  working.  At  this  initial  stage, 
benefits  are  the  most  remote,  inertia  is  strongest,  the  potential  user  has 
nothing  invested  in  the  process,  and  a  variety  of  easier  alternatives  pre¬ 
sent  themselves.  Before  addressing  any  substantive  issues,  the  potential 
user  must  determine  whether  and  how  to  use  the  aid,  learn  or  refresh  the 
necessary  procedural  knowledge,  obtain  appropriate  access  to  the  aid,  and 
follow  the  prescribed  startup  procedures.  Although  this  phase  is  intrin¬ 
sically  an  investment  of  sorts,  it  is  essential  to  minimize  the  effort  and 
stress  involved  and  to  provide  as  much  assistance  and  encouragement  as  pos¬ 
sible.  In  particular,  the  following  goals  can  help  to  overcome  the  initial 
block  to  usage. 

Clear  Product  Identification.  A  decision  aid  should  come  with  a  clear 
label  designating  the  aid's  intended  use,  the  time  and  training  it  requires, 
its  products,  and  a  reference  to  a  further  source  of  information.  If  pos¬ 
sible,  a  hot-line  telephone  should  be  available  for  instant  assistance; 
failing  this,  at  least  one  person  or  office  might  assume  primary  responsi¬ 
bility  for  providing  users  with  information. 

Aesthetic  Package  Design.  The  decision  aid  should  appear  functional 
and  convenient  to  use.  It  should  appear  to  be  an  efficient,  robust,  and 
task-oriented  machine,  solidly  constructed  without  much  unnecessary  clutter. 
The  controls  and  displays  should  appear  as  simple  as  possible  to  avoid  in¬ 
timidating  the  potential  user,  and  all  features  should  be  clearly  labeled. 
Within  these  constraints,  further  efforts  should  focus  on  the  visual  appeal 
of  the  machine  itself,  particularly  the  appeal  of  the  area  the  user  will  be 
occupying . 

Training  and  Documentation .  Although  a  detailed  system  description  and 
a  technical  reference  document  should  be  available  for  reference  or  for 
higher  level  training,  the  ordinary  user  should  need  a  minimum  of  instruc¬ 
tion  and  documentation.  Ideally,  the  system  should  incorporate  a  self- 
contained,  optional  tutorial  rather  than  an  off-line  training  manual;  for 
ordinary  use,  the  startup  instructions  should  be  few  and  simple  enough  to 
appear  on  a  panel  attached  to  the  machine. 

Access  and  Availability.  The  physical  setup  of  the  decision  aid  and 
the  startup  routine's  software  should  be  designed  to  minimize  startup  effort 
A  single  switch  or  control  should  turn  power  on  (if  necessary) ,  and  an  auto¬ 
matic  loading  routine  should  initiate  the  startup  procedure  without  further 
commands.  Any  administrative  procedures  such  as  accounting,  user  identifi¬ 
cation,  and  authorization  should,  if  possible,  be  taken  care  of  at  the  site 
of  the  aid,  with  specially  designed  routines  to  minimize  the  amount  of  paper 
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work  required.  For  example,  instead  of  filling  out  an  authorization  form, 
a  user  might  simply  type  in  a  name  or  identification  code,  allow  the  machine 
to  prepare  the  necessary  documents,  and  sign  the  completed  form. 

User  Participation  in  Customized  Design.  When  an  aid  is  developed  with 
a  particular  user  population  in  mind,  the  opinions  of  representatives  from 
the  user  population  should  be  solicited  and  then  incorporated  in  the  aid's 
design.  Apart  from  the  direct  effect  of  customizing  the  product  to  meet 
unique  needs  and  to  conform  as  much  as  possible  to  current  procedures  and 
conventions,  the  participation  of  the  user  population  will  facilitate  train¬ 
ing  and  help  smooth  the  transition  process.  Most  important,  by  involving 
users  in  the  design  of  the  system,  the  designers  can  overcome  the  users' 
natural  tendency  to  oppose  the  imposition  of  an  unfamiliar  process  by  out¬ 
side  forces,  and  instead  foster  a  feeling  of  personal  investment  in  the 
success  of  the  aid.  The  next  section  of  this  report  considers  in  detail 
the  importance  of  user  involvement  for  successful  implementation. 


Ease  and  Effectiveness  of  Operation 

Once  the  user  has  decided  to  employ  a  decision  aid,  success  depends:  on 
the  aid's  ability  to  hel[  the  user  through  the  procedures  without  disrujtion 
due  to  lapses  in  attention  or  motivation.  In  the  on-going  dialogue,  the 
aid's  outputs  should  be  clear  and  useful,  and  the  required  user  inputs  should 
impose  a  minimum  of  strain  on  the  user.  To  achieve  these  coals,  a  successful 
decision  aid  should  incorporate  simple  control  mechanisms,  continuous  sensori¬ 
motor  feedback,  effective  design  cf  output  formats,  and  human-engineered  pro¬ 
cedural  requirements . 

Simple  Controls.  One  of  the  most  frequent  complaints  of  potential  de¬ 
cision  aid  users  (particularly  high-level  decision  makers)  is  that  they  are 
required  to  type  all  their  inputs  on  a  keyboard.  Although  an  experienced 
typist  or  computer  user  might  feel  comfortable  with  a  keyboard-based  input 
method,  a  less  experienced  tyj  ist  might  experience  impatience  because  of 
limited  typing  speed,  frustration  because  of  high  error  rates,  embarrassment 
in  front  of  onlookers,  and,  in  some  cases,  a  feeling  that  the  use  of  tiie  aid 
is  primarily  a  clerical  chore  rather  than  a  more  challenging  analytic  task. 
Furthermore,  while  concentrating  on  avoiding  typographical  or  spelling  er¬ 
rors,  a  decision  maker  is  distracted  from  the  substantive  task  at  hand.  When 
time  and  organizational  pressures  severely  constrain  the  decision  maker, 
these  effects  may  increase  so  significantly  that  the  aid  is  abandoned. 

This  problem  could  be  avoided  by  using  a  typdst  or  a  specially  trained 
operator  to  enter  the  user's  orally  expressed  responses,  but  such  an  ap¬ 
proach  would  increase  operating  costs  and  personnel  requirements,  would  make 
the  aid  less  accessible,  and,  most  important,  would  eliminate  the  direct 
private  and  personal  involvement  of  the  decision  maker.  Therefore,  the  use 
of  a  clerical  assistant  is  unacceptable  in  most  cases. 

Fortunately,  some  simple  alternate  approaches  can  eliminate  the  need 
for  most  keyboard  inputs.  These  approaches  may  range  from  the  simple  but¬ 
tons  and  levers  that  control  commercial  computer  games  to  futuristic  devices 
such  as  speech  recognizers,  visual  pattern  detectors,  and  pressure-sensitive 
display  screens.  Although  further  human-engineering  efforts  will  be  needed 
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to  determine  the  most  effective  functions,  sizes,  positions,  and  configura¬ 
tions,  these  simplified  input  devices  can  already  produce  big  improvements 
in  user  engineering.  As  more  advanced  technology  becomes  available,  it  will 
become  even  easier  for  the  untrained  user  to  enter  information  and  to  issue 
commands  quickly,  painlessly,  and  reliably. 

Continual  Sensorimotor  Feedback.  When  two  people  interact  in  normal 
conversation,  the  information  transmitted  is  more  than  just  the  words  that 
are  being  uttered.  Hie  "carrier  wave"  that  makes  a  conversation  seem  natural 
is  an  almost  unnoticed  background  of  eye  motions,  nonverbal  noises,  postural 
adjustments,  and  facial  movements,  all  of  which  transmit  such  procedural 
messages  as  "I  am  still  listening,"  "I  have  something  to  say,"  or  "I  am  not 
sure  I  understand  what  you  mean."  Without  this  continuous  nonverbal  inter¬ 
action,  the  conversation  would  become  awkward  and  the  participants  would 
feel  uncomfortable.  Occasional  long  pauses  would  occur,  making  the  partici¬ 
pants  unsure  about  whether  the  dialogue  should  continue.  At  other  times, 
two  or  more  parties  would  attempt  to  speak  at  once,  inducing  not  only  con¬ 
fusion  but  a  certain  amount  of  social  friction  as  well.  The  nonverbal  com¬ 
ponent  of  a  face-to-face  conversation  helps  to  avoid  these  problems,  smooth¬ 
ing  the  social  content  and  assuring  efficient  communication. 

The  same  observation  applies  to  the  interaction  between  the  decision 
aid  and  its  user;  the  aid  must  continually  assure  the  user  that  procedures 
are  in  normal  working  order,  that  inputs  are  being  properly  received,  and 
that  output  responses  are  on  schedule.  To  provide  continuous  or  frequent 
indication  that  all  functions  are  working  properly,  for  example,  the  aid 
may  include  a  clock  whose  display  changes  every  second.  To  provide  immedi¬ 
ate  sensorimotor  feedback  by  acknowledging  every  input  without  perceptible 
delay,  the  aid  may  use  a  visual  signal,  a  simple  tone,  or  an  echo  of  the 
user's  input.  This  feature  is  especially  necessary  when  time  is  of  the  es¬ 
sence  and  the  user  is  preoccupied  with  the  overall  process;  even  a  fraction 
of  a  second  without  response  can  arouse  either  impatience  (in  the  case  of 
experienced  users  who  have  previously  been  led  to  expect  instantaneous  re¬ 
sponses)  or  fear  of  computer  malfunction  (in  the  case  of  inexperienced  users 
who  simply  do  not  know  whether  a  delay  is  normal) .  Whenever  an  especially 
long  delay  (more  than  30  seconds)  is  absolutely  necessary,  the  aid  should 
not  only  acknowledge  the  input  that  initiated  the  long  operation,  but  also 
provide  an  estimate  of  the  time  needed  to  complete  it  and  an  option  to  can¬ 
cel  a  very  long  operation  if  the  user  so  wishes.  During  the  delay,  music 
or  graphical  displays  can  be  used  to  maintain  contact  and  hold  the  user's 
attention . 

Effective  Output  Formats.  The  perceived  value  of  an  aid's  outputs  will 
depend  not  only  on  the  contents  of  the  displays  and  printouts,  but  also  on 
their  format  and  style.  A  well-designed  output  is  easily  read.  It  should 
direct  the  user's  attention  to  the  proper  information,  facilitate  the  use; 's 
ability  to  focus  on  selected  items,  and  all  the  while  remain  aesthetically 
attractive.  The  following  ideas  illustrate  possible  ways  of  achieving  these 
goals . 

•  Replace  textual  alphanumeric  outputs  with  such  formats  as  pictorial 
symbols,  photographic  images,  graphs,  and  maj s . 
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•  Use  prerecorded  or  synthesized  speech  to  present  passages  of  text; 
to  provide  procedural  guidance;  to  annotate  graphs,  charts,  and 
maps;  and  to  add  emphasis  to  important  alphanumeric  display  messages. 

•  Use  unique  nonverbal  sounds  (notes,  chords,  noises)  to  attract  the 
user's  attention  whenever  something  unusual  demands  an  especially 
high  level  of  alertness. 

•  Use  motion  and  color  to  direct  the  user's  visual  attention  to  spe¬ 
cific  segments  of  the  displays. 

•  Use  graphics  and  music  to  enhance  the  aid's  aesthetic  appeal  and 
its  overall  image  of  quality. 


Human  Factors  Engineering.  Even  though  every  individual  task  associ¬ 
ated  with  the  decision  aid  may  be  easily  within  the  user's  grasp,  the  combi¬ 
nation  of  tasks  to  be  accomplished  in  a  short  time  span  may  exceed  the  user's 
limits.  An  approach  using  human  factors  engineering  can  identify  those 
points  where  overloads  of  this  type  are  likely  and  then  indicate  ways  in 
which  the  redesign  of  some  portion  of  the  aid  can  reduce  overload.  For  ex¬ 
ample,  if  the  user  must  attend  to  a  number  of  stimuli  simultaneously,  it  may¬ 
be  possible  to  present  them  through  different  sensory  channels  or  to  present 
them  sequentially  to  avoid  confusion.  Altering  a  display  configuration  or 
an  input  device  may  have  a  significant  impact  on  the  amount  of  strain  imposed 
and  improving  timing  and  sequencing  may  allow  users  to  work  more  effectively 
without  reducing  the  actual  task  requirements.  Finally,  designing  specific 
color-coding  schemes,  symbols,  and  auditory  cues  to  correspond  with  the  us¬ 
er's  "natural"  expectations  may  relieve  some  of  the  effort  involved  in  inter¬ 
preting  outputs. 


Minimal  Requirements  for  Technical  Knowledge.  Although  it  may  be  impos¬ 
sible  to  make  an  aid's  decision-analytic  techniques  completely  transparent  to 
the  user,  every  effort  should  be  made  to  minimize  the  requirements  for  spe¬ 
cialized  methodological  training.  In  no  case  should  a  technical  decision- 
analytic  term  be  used  without  explanation  of  its  specialized  meaning;  if  at 
all  possible,  technical  jargon  that  might  intimidate,  confuse,  or  alienate 
the  user  should  be  avoided  altogether.  If  analytic  methods  must  be  referred 
to,  it  might  be  preferable  to  invent  new  terms  rather  than  risk  the  confusion 
that  might  arise  from  using  ambiguous  decision-analytic  terms  such  as  utility  - 
at  tribute ,  risk  ,  weight ,  and  option .  If  the  goal  is  to  communicate  wit,i  a 
naive  user,  there  is  little  reason  to  insist  on  traditionally  accepted  terms. 
Of  course,  if  the  analysis  can  be  conducted  at  the  level  of  direct  judgments 
(such  as  binary  choices) ,  keeping  the  decision-analytic  implications  of  the 
user's  responses  internal  to  the  aid's  program,  so  much  the  better. 


Intermediate  Reinforcement  to  Increase  Attention  and  Motivation 


The  preceding  discussion  has  considered  ways  to  attract  the  user  to  the 
aid  and  to  simplify  overall  operation  of  the  aid.  However,  because  the  user 
is  a  human  being  with  a  limited  attention  span  and  other  responsibilities 
competing  for  time  and  attention,  the  aid  must  do  more  than  just  smooth  the 
path  toward  the  ultimate  goal.  In  addition,  the  aid  must  help  the  user  to 
follow  that  path  without  losing  sight  of  the  goal  or  being  distracted  alone; 
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the  way.  Because  the  successful  use  of  a  decision  aid  requires  a  long  chain 
of  behaviors,  many  of  which  will  be  quite  unexciting,  the  user  will  face 
boredom,  fatigue,  and  impatience  at  times  when  the  goal  still  appears  quite 
remote.  Insofar  as  possible,  the  aid  should  counter  those  effects  by — 

•  reminding  the  user  of  the  ultimate  goal, 

•  providing  milestones  and  progress  reports  along  the  way, 

•  rewarding  the  user  for  completing  intermediate  goals, 

•  timing  and  sequencing  tasks  to  avoid  boredom,  and 

•  reinforcing  the  user  for  maintaining  a  high  level  of  attention. 

Goal  Focusing.  One  useful  way  to  keep  the  decision  maker  aimed  toward 
the  desired  destination  is  to  provide  a  sort  of  road  map  in  the  form  of  a 
milestone  chart.  This  not  only  reminds  the  user  of  the  ultimate  goal,  but 
provides  a  set  of  more  modest  subgoals  for  the  user  to  complete.  As  each 
suogoal  is  reached,  it  can  be  represented  on  a  progress  chart,  thereby  re¬ 
warding  the  decision  maker  while  pointing  toward  the  next  task.  The  comple¬ 
tion  of  a  subgoal  might  be  a  good  occasion  for  a  break  or  for  a  review  of 
the  partial  results  available.  These  results,  in  the  form  of  hard-copy 
charts,  graphs,  tables,  and  text,  can  act  as  a  further  reward  by  providing 
the  user  with  valuable  information  and  tangible  evidence  of  work  completed. 

Timing,  Sequencing,  and  Variation.  No  matter  how  easy  or  enjoyable  a 
task,  it  will  eventually  lose  the  user's  interest  if  it  is  too  prolonged  oi 
repeated  too  often.  Satiation  with  task  rewards,  habituation  ‘  the  visual 
and  aural  stimuli  presented,  and  general  fatigue  will  increase  'til  motiva¬ 
tion  drops,  attention  lags,  and  error  rates  rise. 

By  dividing  the  overall  task  into  shorter  segments  and  varying  succes¬ 
sive  tasks  (e.g.,  using  different  sensory  modalities,  different  display  col¬ 
ors,  different  muscular  movements)  ,  the  decision  aid  can  keep  the  user  more- 
attentive  and  better  motivated.  Human  engineering  can  achieve  the  right 
balance  between  the  attentional  benefits  to  be  gained  from  shorter  tasks 
and  the  possible  confusion  and  delay  involved  in  switching  tasks  too  fre¬ 
quently.  Further  study  might  identify  groups  of  complementary  tasks  that 
could  be  effectively  organized  into  a  recurring  cycle,  to  provide  the  neces¬ 
sary  variety  without  unnecessary  shifts  in  attention.  Ideally,  the  transi¬ 
tion  from  task  to  task  should  be  significant  and  frequent  enough  to  prevent 
boredom  and  fatigue,  yet  smooth  and  logical  enough  to  maintain  continuity. 

Direct  Reinforcement  for  Task-Oriented  Behavior.  Although  some  moti¬ 
vation  may  result  simply  from  attaining  intermediate  goals,  this  source  of 
reinforcement  can  be  simply  and  directly  augmented  by  providing  more  direct 
rewards  as  well.  At  the  end  of  a  given  task  sequence,  the  user  might  be 
given  the  opportunity  to  clear  his  or  her  mind  by  engaging  in  some  soil  of 
recreational  activity  for  a  limited  time.  The  aid  might,  for  example,  ;  ro- 
vide  a  choice  between  a  video  game,  a  passage  of  recorded  music,  a  selection 
of  puzzles  or  jokes,  and  a  display  of  comp'uter  art.  In  order  to  control  the 
amount  of  time  spent  on  such  extraneous  pursuits,  access  to  the  reward  ac¬ 
tivities  might  be  programed  to  occur  only  at  random  times,  contingent  upon 
successful  completion  of  subqoals.  Behavioral  research  has  shown  that  random 
reinforcement  of  this  sort  is  often  far  more  efficient  at  maintaining  effort 
than  regular  schedules  of  reinforcement  with  the  same  overall  frequency  of 
reward . 
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A  further  way  of  reinforcing  attention  during  the  performance  of  a  task 
(without  distracting  the  user  from  the  task  itself)  might  be  to  measure  the 
user's  response  times,  providing  feedback  in  the  form  of  occasional  perfor¬ 
mance  reviews  (to  be  presented  at  the  task's  completion)  and  bonus  rewards 
for  good  performance.  For  example,  a  numerical  alertness  score  based  on  the 
user's  speed  of  response  might  be  combined  with  an  error  rate,  or  othei  be¬ 
havioral  measure,  to  determine  the  likelihood  of  a  reward  at  the  end  of  each 
task  segment.  Auditory  feedback  might  be  useful  for  this  function  in  much 
the  same  way  that  the  bells,  clicks,  and  various  electronic  sounds  reinforce 
the  users  of  video  games  and  pinball  machines  without  impairing  their  atten¬ 
tion.  Because  preferences  and  needs  in  this  area  may  vary  widely  from  user 
to  user,  the  ability  to  involve  users  in  the  initial  design  (discussed  u, 
the  next  section  in  this  report)  or  to  adapt  procedures  to  individual  needs 
(discussed  later  in  this  section)  will  be  especially  lrajortant. 


Final  Products 


Once  the  user  has  completed  the  analysis,  the  decision  aid  shield  pro¬ 
vide  as  much  reinforcement  as  possible  in  order  to  make  its  use  more  attrac¬ 
tive  in  the  future,  while  continuing  to  offer  whatever  sup  port  is  available 
to  translate  the  results  of  the  analysis  into  action.  The  re  infer comer,  t 
might  include  the  following: 

•  hard  copies  of  tables,  charts,  graphs,  and  other  materials  that 
might  be  useful  to  brief  others  on  the  outcome  of  the  analysis 
(or  to  inp.ut  into  some  luqher  order  decision  p  rocess)  ; 

•  a  [reformatted  report  that  presents  the  analytic  results  and  ra¬ 
tionale  in  a  readable  format,  along  with  suj.-jort  inq  document  at  icn 
on  the  analytic  methods  used  and  the  conclusions  reached;  or, 
perhap.s , 

•  a  printed  or  videotap  ed  p  rotocol  of  the  entire  session,  includir.  : 
a  visual  record  of  what  has  appeared  on  the  display  screen  and  an 
audio  or  textual  record  of  inputs  and  verbal  outputs. 

Further  assistance  might  take  the  form  of  follow-on  analysis  routines . 
For  example,  once  the  user  has  selected  an  overall  course  of  action,  the  aid 
might  offer  an  option  to  help-  construct  a  more  detailed  implementation  1  la:.. 
If  sensitivity  analyses  indicate  the  need  for  better  data  on  some  critic..  1 
topics ,  a  value  ot-i nformation  analysis  might  help  determine  which  data  tv- 
collect  and  how  extensive  an  effort  is  needed.  Similarly,  if  a  short-range 
decision  has  been  made,  the  aid  might  provide  some  help,  toward  lnteorat i no 
it  with  the  related  mid-to-long-range  considerations. 

Finally,  whatever  reinforcements  were  available  up-on  completion  ot  the 
subgoals  ought  to  be  presented  (wi th  certainty  and  in  greater  quantity)  when 
all  the  session's  work  has  been  completed.  Summary  feedback  on  the  user  's 
behavioral  data  (response  times,  error  rates,  etc.)  might  be  useful  foi  the 
user's  own  benefit,  although  care  should  be  taken  to  preserve  the  usei's 
confidence  in  the  privacy  of  this  information.  Unless  time  is  extremely 
short,  the  user  should  be  permitted  to  enpoy  the  recreational  rewards  and 
the  satisfaction  of  having  finished  the  comp  let c  analysis.  Most  important  , 
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the  aid  should  acknowledge  the  user's  hard  work  and  elicit  any  comments, 
suggestions,  or  questions  that  might  help  to  improve  future  versions  of  the 

aid . 


Adapting  Procedures  to  Meet  Individual  Needs 

If  the  suggestions  specified  earlier  are  all  implemented,  the  resulting 
decision  aid  will  be  well  engineered  in  terms  of  an  overall  user  population. 
However,  since  the  aid's  usage  is  based  on  the  behavior  of  several  individ¬ 
ual  users  rather  than  a  single  group,  the  ability  to  fine-tune  the  aid  to 
individual  specifications  will  dramatically  improve  its  acceptance.  The 
more  variable  individual  users  (or  individual  problems)  are,  the  more  impor¬ 
tant  this  customization  will  become. 

Design-to-Time  Control  of  Processes.  Perhaps  the  most  critical  varia¬ 
tion  from  problem  to  problem  is  the  amount  of  time  the  user  can  afford  to 
spend  performing  an  analysis.  For  high  stakes  (complex  decisions  for  which 
time  is  not  a  factor) ,  the  user  would  like  to  ensure  maximum  validity  and 
completeness,  even  at  the  expense  of  a  longer,  more  extensive  analysis;  this 
might  entail  a  variety  of  sensitivity  analyses,  consistency  checks ,  data 
searches,  and  other  procedures.  At  the  opposite  extreme,  if  a  decision  must 
be  made  immediately  based  only  on  whatever  information  is  in  the  decision 
maker's  head,  any  effort  to  check  for  methodological  correctness  may  be  per¬ 
ceived  as  an  unnecessary  waste  of  valuable  time.  Similar  variety  in  users' 
preferences  may  stem  from  the  decision  makers'  personalities,  from  organi¬ 
zational  factors  that  influence  the  aid's  availability  and  usage,  and  from 
the  urgency  of  other  tasks  competing  for  the  decision  makers'  time. 

Adaptability  to  Various  Training  Levels.  One  universal  problem  with 
multiple-user,  interactive  computer  programs  is  the  need  to  accommodate  a 
variety  of  skill  and  training  levels.  If  an  aid  is  self-explanatory  enough 
to  permit  error-free  use  with  a  complete  novice  at  the  controls,  it  will 
very  likely  move  far  too  slowly  for  a  more  experienced  user.  But  if  the 
aid  is  faster  and  more  streamlined  (e.g.,  requiring  only  abbreviated  com¬ 
mands  instead  of  complete  words),  it  is  more  likely  to  cause  confusion  and 
error  in  a  novice. 

Because  a  decision  aid  of  the  sort  discussed  here  should  be  designed 
for  a  wide  variety  of  user  skills  (it  must  satisfy  a  number  of  naive  users, 
but  should  also  cultivate  "repeat  customers"),  one  useful  approach  might  be 
to  provide  three  tracks: 

1.  A  novice  level  for  the  first-  or  second-time  user.  This  might 
include  a  brief  tutorial  in  the  aid's  procedures,  very  explicit 
user  instructions  with  accompany ing  examples  ,  as  natural  a  mode 
of  interaction  as  possible,  and  an  analytic  capability  restricted 
to  a  core  of  basic  procedures. 

2.  A  standard  level  for  the  occasional  user.  If  the  user  is  experi¬ 
enced  enough  so  that  the  benefits  of  the  novice  level  are  no 
longer  worth  the  extra  time  required,  a  more  streamlined  apj  roach 
might  be  more  effective  and  might,  add  analytic  features  beyond 
the  basic  novice  repertoire. 
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3.  An  expert  level  for  the  experienced,  frequent  user.  This  level 
would  extend  the  range  of  analytic  capabilities,  would  emphasize 
speed  and  efficiency  rather  than  error  protection,  and  might 
give  the  user  control  of  certain  performance  parameters  (e.g., 
response  modes,  frequency  of  reinforcement,  output  formats,  speed, 
precision  trade-offs)  to  suit  individual  needs. 

It  should  always  be  possible  for  a  user  to  change  tracks  at  any  point 
in  the  analysis,  either  permanently  or  temporarily,  without  jeopardizing  ex¬ 
isting  results.  A  help  button  or  instruction  could  be  used  to  inform  the 
user  in  more  detail  about  the  options  available  at  any  point.  A  more  sophis¬ 
ticated  aid  might  keep  a  record  of  a  decision  maker’s  past  usage  and  perfor¬ 
mance  (e.g.,  error  rates,  speed  of  response),  automatically  starting  the 
user  at  the  most  appropriate  level  and  modifying  the  level  based  on  current 
performance  (but  always  subject  to  user  override) . 

Adaptation  to  User's  Personal  Preferences.  Once  a  user  has  a  certain 
degree  of  familiarity  with  the  aid,  it  may  be  desirable  to  make  minor  adjust¬ 
ments  and  alterations  in  order  to  accommodate  the  user's  individual  prefer¬ 
ences  or  to  comply  with  a  specific  set  of  standard  conventions.  For  example, 
input -output  choices  of  the  color-coding  scheme,  symbology,  and  display  for¬ 
matting  might  initially  be  given  arbitrary  default  settings,  but  on  the  us¬ 
er's  request,  they  might  be  altered  to  fit  individual  needs.  Similarly, 
operational  features  such  as  the  mode  of  input  or  the  machine's  average  time 
to  react  may  need  to  be  adjusted  (as  in  the  case  of  some  computer  chess¬ 
playing  programs,  in  which  a  delay  was  added  because  users  felt  uncomfortable 
with  the  instantaneous  responses  the  machines  had  been  making) . 

A  more  sophisticated  approach  to  customization  would  have  the  aid’s  rou¬ 
tines  expressed  as  functions  of  several  parameters,  each  of  which  might  cor¬ 
respond  to  some  aspect  of  the  user's  skills  and  preferences .  The  frequent 
user  could  initiate  a  questionnaire  routine  that  would  replace  the  default 
settings  for  all  of  these  parameters  with  user-specified  values  (e.g.,  "How 
good  a  typist  are  you?"  "Which  of  these  type  faces  do  you  prefer?"  "Tn 
general,  which  is  more  important  to  you,  speed  or  completeness? ") .  Then,  a 
special  version  of  the  aid's  routines  could  be  compiled  using  the  profile's 
values.  As  those  values  changed,  the  user  could  modify  the  profile  and  alter 
the  routines  accordingly. 

The  methods  just  discussed  would  require  a  fairly  sophisticated  user; 
a  novice  or  occasional  user  would  not  be  sensitive  enough  to  minor  alterations 
to  make  the  effort  of  fine-tuning  worthwhile.  However,  a  very  so; histicuted 
version  of  the  machine  might  automatically  select  which  parameters  each  user 
could  adapt,  basing  its  selection  on  physiological  monitoring,  if  available, 
and  on  the  user's  behavioral  state,  inferred  from  response  time,  error  rates, 
and  answers  to  direct  inquiries  (e.g.,  "Do  you  want  to  continue  or  would  yc u 
like  a  break?").  Data  about  behavioral  state  could  be  used  to  check  for 
user  alertness,  to  regulate  the  frequency  of  breaks  and  reinforcements,  and 
to  adjust  system  parameters  experimentally  in  order  to  imj  rove  user 
performance . 
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Conclusion 


In  the  future,  methods  of  computer-aided  decision  making  may  bear  litt.c 
resemblance  to  the  methods  available  today.  As  the  general  public  becomes 
more  knowledgeable  about  computers,  and  as  computer  usage  by  nonspecialists 
becomes  widesptcad,  some  of  the  blocks  that  have  been  the  target  of  the  cur¬ 
rent  efforts  may  disappear  (as  others  arise) .  Also,  as  available  technol- 
ogy — both  hardware  and  software — becomes  cheaper,  more  accessible,  and  more- 
sophisticated,  more  ambitious  goals  will  become  feasible.  Speech  recogni¬ 
tion,  natural  language  comprehension,  visual  image  perception,  three- 
dimensional  displays,  and  even  more  advanced  features  will  someday  be  com¬ 
monplace.  However,  only  by  working  now  to  pioneer  useful-  applicati< -ns  can 
we  hope  to  influence  the  course  of  such  developments  and  find  a  market  for 
them  when  they  are  ready.  The  issues  discussed  in  this  section,  an.  the  im¬ 
plementation  features  recommended,  will  provide  a  sound  basis  for  d< cision 
aid  engineering  in  the  near  term  and  a  guide  for  the  eventual  incorporation 
of  future  technology. 


INVOLVING  USERS  IN  THE  DEVELOPMENT  OF  DECISION-ANALYTIC  AIDS: 

THE  PRINCIPAL  FACTOR  IN  SUCCESSFUL  IMPLEMENTATION 

Over  the  past  25  years,  hundreds  of  scientific  studies  of  human  judgment 
and  decision  making  have  reached  one  basic  conclusion:  Unaided  human  judg¬ 
ment  has  limitations.*  As  a  result  of  these  findings,  as  well  as  of  advances 
in  normative  decision  theory  (e.g.,  von  Neumann  &  Morgenstern,  1947;  Savage, 
1954;  Raiffa,  1968;  Keeney  &  Raif fa ,  .1976)  and  computer  technology,  judgment/ 
decision  researchers  have  begun  developing  computer-based  decision-analytic 
aids  to  help  decision  makers  improve  and  extend  their  cognitive  ability. 

These  aids  include  different  types  of  multi-attribute  utility  assessment  pro¬ 
grams,  such  as  HIVAL  (Allardyce  &  Peterson,  1979)  and  POLICY  (Hammond,  Cook, 

&  Adelman,  1977),  as  well  as  traditional  decision-analytic  aids  requiring 
probability  and  utility  assessment,  such  -as  INFER  (Amey,  Feuerwerger  &  Gu- 
lick,  1979a)  and  OPINT  (Amey,  Feuerwerger,  &  Gulick,  19  79b).  Such  aids  have- 
been  used  successfully  in  a  wide  range  of  settings,  as  indicated  in  compendi- 
ums  by  Kaplan  and  Schwartz  (1977)  ,  by  Keeney  and  Raiffa  (1976)  ,  and  by  Kelly 
(1979). 

On  the  basis  of  previous  success,  one  can  expect  increased  utilization 
of  computer-based  decision-analytic  aids  with  stand-alone  capabilities  for 
routine  use  by  internal  analysts  and  decision  makers  without  outside  consulta¬ 
tion.  The  thesis  of  this  section  is  that  decision-analytic  aids  will  seldom 
achieve  a  stand-alone  status  unless  eventual  users  are  involved  in  their  de¬ 
velopment.  Ihe  term  users  applies  here  both  to  the  persons  running  the  de¬ 
cision  aid  and  to  the  decision  makers  utilizing  its  results.  Ihe  previous 
section  focused  on  the  interface  between  the  aid  and  its  hands-on  users ,  who 
may  or  may  not  be  decision  makers.  The  position  was  that  the  better  the 
general  behavioral  characteristics  of  the  aid,  the  higher  the  motivation 
of  the  hands-on  user  and  therefore  the  greater  the  probability  that  the  aid 


*The  interested  reader  is  referred  to  Hammond,  McClelland,  and  Mumpower 
(1980);  Slovic,  Fischhoff,  and  Lichtenstein  (1977);  and  Slovic  and  Lichten¬ 
stein  (1971)  for  reviews  of  this  research. 
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will  be  successfully  integrated  into  the  organization.  In  this  section,  the 
focus  is  on  the  interface  between  the  user  (and  decision  aid)  and  the  larger 
decision-making  org .nization .  The  position  here  is  that  successful  general 
behavioral  characteristics  of  decision  aids  are  necessary,  but  not  suffi¬ 
cient,  for  aid  implementation.  In  addition,  the  involvement  of  decision 
makers  in  aid  design  is  essential  for  implementation,  for  this  involvement 
develops  the  understanding  and  commitment  necessary  for  implementing  a  dif¬ 
ferent  decision-making  approach  and  tailors  the  characteristics  of  the  aid 
to  the  users'  needs  within  their  organizational  context. 

The  judgment/decision  research  literature  has  not  emphasized  that  user 
involvement  in  aid  design  is  important  for  successful  aid  implementation. 
Support  for  this  position  comes  primarily  from  two  sources:  (1)  the  recent 
systematic  evaluation  of  an  experimental  decision-analytic  aid  developed  fo: 
use  by  the  Operations  Directorate  of  the  Joint  Chiefs  of  Staff  and  (2)  tine- 
history  of  model  implementation  in  operations  research  and  management  sci¬ 
ence.  Although  the  need  for  user  involvement  appears  obvious,  these  sources 
indicated  that  user  involvement  is  often  neglected  in  the  development  of 
analytical  decision  aids,  frequently  resulting  in  unsuccessful  implementa¬ 
tion  efforts  . 


Evaluation  of  Decision-Analytic  Aids 

R-SCREEN .  Saqe  and  White  (I960)  recently  evaluated  a  multi-attribute 
utility  assessment  (MAUA)  aid,  called  R-SCREEN  (Rapid  Screening  of  Decision 
Options) .  The  R-SCREEN  aid  was  developed  for  use  by  operational  analysts 
in  the  Joint  Operations  Division  (JOD)  within  the  Operations  Directorate  cf 
the  Office  of  the  Joint  Chiefs  of  Staff. 

Funding  for  R-SCREEN  was  provided  by  an  agency  that  is  tasked  with  moni¬ 
toring,  evaluating,  and  improving  the  overall  information  flow  within  the 
World  Wide  Military  Command  and  Control  System  in  support  of  information  re¬ 
porting,  information  analysis,  decision  making,  and  information  dissemina¬ 
tion.  Specifically,  R-SCREEN  was  developed  to  support  the  option  generation 
and  selection  process  as  it  occurs  in  command  centers  in  crisis  situations. 

JOD  decision  makers  used  R-SCREEN  by  implementing  four  steps.  First, 
they  selected  from  among  three  prestructured  templates  (or  hierarchies) 
the  template  most  appropriate  for  the  particular  problem  at  hand,  and  they 
made  minor  modifications  to  the  structure  as  needed  to  match  the  template 
to  the  criteria  most  relevant  to  the  particular  problem.  Second,  they  iden¬ 
tified  various  alternative  courses  of  action  for  evaluation.  Third,  they 
scored  each  of  the  alternative  courses  of  action  on  each  of  the  lower  level 
attributes  and  then  assessed  criterion  importance  weights  (essentially  usnw 
Edwards's  (1977)  ratio  estimate  technique)  in  order  to  determine  the  rela¬ 
tive  utility  of  each  alternative.  And  fourth,  they  assessed  the  sensitivity 
of  the  analytical  results  by  evaluating  the  impact  of  changing  utility 
scores  and  criterion  weights. 

R-SCREEN  was  introduced  into  the  JOD  in  spring  1979.  Decision  analysts 
briefed  the  JOD  staff  on  how  to  use  the  aid,  developed  a  user's  guide  sjk?- 
cifically  for  the  aid  (Gulick  &  Allardyce,  1979)  ,  provided  on-the-}ob  train¬ 
ing  sessions  throughout  the  course  of  the  experimental  period,  and,  in 
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general,  made  themselves  immediately  available  at  the  request  of  JOD  person¬ 
nel  to  discuss  R-SCREEN's  utilization.  JOD  personnel  were,  however,  not  in¬ 
volved  in  R-SCREEN's  development  The  exj  mental  period  lasted  approxi¬ 
mately  6  months . 


Sage  and  White  (1980)  evaluated  R-SCREEN  by  the  following  three  proce¬ 
dures:  (1)  informal  interviews  with  JOD  personnel  and  others  familiar  wit), 

the  JOD  operational  environment,  (2)  study  of  various  written  documentation , 
and  (3)  detailed  analysis  of  questionnaire  responses  as  well  as  follow-up 
interviews  with  Pentagon  personnel  and  with  a  group  of  senior  military  and 
civilian  students  from  the  Industrial  College  of  the  Armed  Forces,  who  wer ■■ 
asked  to  evaluate  the  aid  in  an  experimental  context.  S.age  and  White  organ¬ 
ized  the  evaluation  responses  into  the  criteria  and  subcriteria  shown  in 
Table  1.  Although  Sage  and  White  discussed  the  implications  of  the  responses 
in  terms  of  each  of  the  15  subcriteria,  they  did  not  give  R-SCREEN  an  ex¬ 
plicit  score  on  each  criterion.  In  order  to  shorten  this  presentation, 

Table  1  shows  an  overall  score  (  +  ,  -,  or  ?)  based  on  Sage  and  White's  quali¬ 
tative  evaluation  on  each  subcriterion. 


R-SCREEN  ratedi  extremely  well  in  performance  objective  achievement 
(category  1)  and  efficacy  (category  3) .  These  high  ratings  provide  empiri¬ 
cal  support  for  the. claims  of  judgment  and  decision  researchers  who  have 
argued  that  decision-analytic  aids  facilitate  clear  thinking,  educate  de¬ 
cision  makers  about  their  problems,  and  facilitate  communication  (e.g.,  see 
Hammond  et  al . ,  1980) . 

R-SCREEN  rated  poorly,  however,  in  behavioral  criteria  (category  2). 
R-SCREEN  received  a  questionable  rating  on  implementability  because  of  par¬ 
ticipants'  reservations  concerning  its  usefulness  in  a  crisis  management 
environment.  These  reservations  relate  directly  to  R-SCREEN's  ratings  on 
political  acceptability  and  institutional  constraints.  Sage  and  White  dis¬ 
cuss  these  two  subcriteria  as  follows: 


Political  acceptability : 

Political  issues  were  viewed  by  several  subjects  as  potential 
barriers  to  acceptance  of  systemic  aids,  such  as  R-SCREEN,  into 
an  operational  environment.  Lack  of  senior  level  receptivity 
and  the  personal  decision-making  styles  of  flag  officers  were 
seen  as  potential  hindrances.  Full  management  and  other  lead¬ 
ership  commitment  to  implementation  testing  of  decision  aids 
were  viewed  as  very  necessary.  Significant  barriers  to  accep¬ 
tance  of  an  aid  were  felt  to  result  with  the  absence  of  this 
commitment . 

Institutional  constraints: 


Questionnaire  responses  indicated  a  concern  that  R-SCREEN  does 
not  directly  address  the  needs  of  the  JOD,  is  not  particularly 
well  matched  to  the  behavioral  characteristics  of  the  opera¬ 
tional  environment,  may  not  enhance  information  flow,  and  does 
not  possess  desirable  time  to  use  response  characteristics  for 
typical  JOD  operations.  (1980,  p.  0.11) 


« 


15 


Table  1 


Criteria  for  Decision  Aid  Evaluation  and  Evaluation  Scores  for  E-SCREEN 


r.A;:.  : 


Algorithmic  effectiveness  or  per fcmance  objective  achievement 

Logical  soundness 
Improved  decision  quality 
Decision  process  changes 

Behavioral  or  human  factors 

Political  acceptability 
Institutional  constraints 
Impl emen tabi 1 i ty 
Procedural  changes 
Side  effects 

Efficacv 


Time  requirements 

Leadership  and  training  requirements 

Comrr.u  n  i  c  a  t  i  o  r.  accomplishments 

Educational  accorp  1 ishments 

Documentation 

Reliability 

Convenience  of  access 


Note +  means  performed  well;  -  means  performed  poorly;  ?  means  performed 
well  and  poorly  on  questions  comprising  the  subcriterion  category. 


In  short,  R-SCREEN  was  not  tailored  to  the  personal  needs  and  oraar.izat ier..«  1 
context  of  its  eventual  users.  As  a  result,  evaluation  responses  indicated 
that  its  implementation  in  JOD  was  questionable--and ,  in  fact,  it  has  net 
been  implemented  to  date  even  though  respondents  believed  it  would  in;  rove 
decision  quality,  just  as  its  designers  had  claimed. 

It  is  important  to  contrast  the  above  unsuccessful  implementat ion  effort 
with  a  successful  one  in  order  to  gain  insight  into  the  extent  to  which  im¬ 
plementation  is  enhanced  by  tailorinq  a  decision-analytic  aid  to  the  personal 
needs  and  organizational  context  of  its  eventual  users.  The  authors,  however, 
are  not  aware  of  any  evaluation  of  a  decision-analy tic  aid  in  its  operation.!] 
context  that  is  as  systematic  and  thorough  as  the  evaluation  conducted  by 
Sage  and  White.  Although  post  hoc  evaluations  of  successful  implementat l.'n 
efforts  are  open  to  charges  of  bias,  such  an  evaluation  is  presented  briefly 
in  an  effort  to  help1  readers  evaluate  the  adequacy  of  the  thesis  advanced  m 
this  section. 


It. 


MCCRESSA.  The  L'.S.  Marine  Corps,  as  well  as  other  services,  has  a  con¬ 
tinuing  problem  in  assessiny,  under  peacetime  conditions,  the  combat  readi¬ 
ness  of  combat  units.  The  problem  is  compounded  by  the  many  heterogeneous 
attributes  that  are  used  to  describe  the  performance  of  individual  combat 
units  and  by  the  many  criteria,  both  objective  and  subjective,  that  are  com¬ 
monly  used  by  force  commanders  to  define  a  successful  level  of  combat  readi¬ 
ness.  Historically,  there  has  been  almost  no  acceptable  standardization  or 
formalization  of  the  process  of  combat  readiness  evaluation  or  validation 
of  evaluation  results. 

In  support  of  the  Marine  Corps  Combat  Readiness  Evaluation  System; 
(MCCRES),  an  MAUA  aid  called  Marine  Corps  Combat  Readiness  Evaluation  Sys¬ 
tem  Software  Application  (MCCRESSA)  was  developed  (Allen  &  Allardyce,  1976). 
The  Marine  Corps  successfully  tested  MCCRESSA  in  an  operational  settinc  m 
August  1977  and  the  aid  is  now  in  routine  use  throughout  the  Marine  Corps. 

MCCRESSA  and  R-SCREEN  are  extremely  similar.  Both  were  designed  from, 
the  same  generic  MAUA  software.  If  anything,  R-SCREEN  is  more  sophisticated 
analytically  than  MCCRESSA  because  it  forces  the  user  to  assign  criterion 
weights  moving  from  the  bottom  to  the  top  of  the  hierarchy,  thereby  ensuring 
that  the  upper  level  weights  are  determined  by  the  scores  on  the  lower  level 
attributes  and  not  by  the  user's  general  perception  of  the  relative  importance 
of  the  upper  level  attributes.  Both  aids  were  designed  to  have  stand-alone 
capabilities.  Yet,  MCCRESSA  was  successfully  implemented  and  R-SCREEN  was 
not . 


MCCRESSA  was  successfully  implemented  because  its  eventual  users  were 
involved  throughout  the  entire  process  of  development  and  implementation . 

The  decision  aid  analysts  worked  directly  with  the  five  Marine  colonels 
tasked  with  developing  and  implementing  MCCRES  over  a  1-year  period.  These 
men  decided  on  the  criteria,  hierarchical  structure,  and  weights  in  the  MAUA 
model  within  MCCRESSA.  They  decided  how  inputs  to  MCCRESSA  would  be  made 
during  actual  MCCRES  evaluations.  They  decided  on  the  type  of  outputs 
MCCRESSA  had  to  provide,  and  the  constraints  under  which  these  outputs  would 
have  to  be  provided,  within  their  operational  context. 

After  a  prototype  aid  was  developed,  the  Marine  colonels  chaired  a  2-day 
conference  for  all  field  commanders  who  would  participate  in  MCCRES  evalua¬ 
tions.  They  showed  the  commanders  how  MCCRESSA  would  be  used  during  each 
evaluation  and  gave  them  an  opportunity  to  ask  questions,  raise  concerns, 
and  suggest  ways  of  better  tailoring  MCCRESSA  to  the  evaluation  process. 

The  colonels  also  went  to  each  of  the  Marine  bases,  where  evaluations  were 
held  to  answer  questions  and  obtain  suggestions  from  personnel  who  would 
actually  use  MCCRESSA  during  an  evaluation.  Some  of  the  lower  level  attri¬ 
butes  in  the  MAUA  hierarchy  and  some  of  the  procedures  for  using  MCCRESSA 
were  modified  on  the  basis  of  the  concerns  and  suggestions  raised  during  the 
conference  and  tour.  There  were  additional  minor  modifications  of  MCCRESSA 
after  its  initial  application  during  some  MCCRESS  evaluations. 

In  sum,  user  involvement  throughout  development  and  implementation  en¬ 
sured  that  MCCRESSA  was  tailored  to  the  Marine  Corps's  needs  and  organiza¬ 
tional  context.  The  authors  believe  this  to  be  the  principal  factor  in 
MCCRESSA's  successful  implementation  in  the  Marine  Corps. 
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1  mp  loiTii : : 1 i;.g  Op ‘erations  Research  Models 

Although  there  are  distinct  differences  between  decis  ion-arialy  tic  aid:; 
and  operations  research  models,  both  represent  highly  analytic  technique.-:  f 
assisting  the  process  and  quality  of  decision  making  within  large  organiza¬ 
tions  .  Consequently,  both  face  similar  implementation  problems.  The  o;  era 
tions  research  ICR)  'management  science  (MS)  literature  over  the  past  two 
decades  has  (1)  documented  numerous  cases  in  which  clients  have  not  used 
analytically  rigorous  OR  models  developed  for  them,  (2)  tried  to  ex;  lair, 
this  phenomenon,  and  (3)  offered  suggestions  for  minimizing  unsuccessful 
implementations.  This  section  briefly  reviews  this  literature. 

Ginzberg  (1978)  divided  the  OR/MS  literature  on  implementation  into  tw 
types:  the  normative  approach  (e.g.,  Ackoff,  1960;  Argyris,  1971;  Grayson, 

1973)  and  the  factor  approach  (e.g.,  Drake,  1971;  Powers  &  Dickson,  1973; 
Rubenstein,  Radner,  Baker,  Heiman,  &  McColly,  1967). 

The  normative  approach  is  based  on  the  field  experience  of 
a  number  of  MS  researcher./'practi  tioners .  These  researchers  typ  i- 
cully  looked  back  at  one  or  more  cases  they  were  involved  in  where 
there  was  substantial  implementation  difficulty,  and  attempted  to 
draw  from  these  experiences  the  general  nature  of  implementation 
problems  and  their  solutions.  Looking  at  this  literature  in  ag¬ 
gregate,  wo  find  substantial  disagreement  on  just  what  the  solu¬ 
tion  to  implementation  problems  should  be....  The  next  develop¬ 
ment  in  implementation  research  was  the  factor  approach.  Each 
factor  study  begins  by  identifying  a  group  of  variables  poten¬ 
tially  relevant  to  implementation  outcomes.  Data  are  then  col¬ 
lected  from  a  sample  of  MS  implementation  projects  -  some  suc¬ 
cessful  and  others  not  -  and  are  used  to  assess  the  relative 
importance  cf  the  different  variables  (or  factors)  to  implementa¬ 
tion  outcomes.  The  results,  however,  are  rather  disappointing. 

Few  general  guidelines  have  emerged  from  this  research,  the  re¬ 
sults  of  different  studies  being  contradictory  in  a  number  of 
cases.  The  only  result  which  is  firmly  established  by  this  re¬ 
search  is  the  important- e  of  management  support  and  user  involve¬ 
ment  to  the  successful  implementation  of  MS/MIS  projects. 

[emphasis  added]  (Ginzbera,  1978,  pp- .  57-58) 

The  research  by  Lonnstedt  (1975)  and  Shycon  (1977),  which  was  not  cite 
by  Ginzberg  (1978)  ,  further  supports  Ginzberg ’s  conclusion.  Lonnstedt 
(1975)  interviewed  key  operations  p-ersonnel  in  12  conpanies,  each  with  its 
own  OR  division,  listed  in  the  Stockholm  Stock  Exchange  in  an  effort  to 
identify  factors  related  to  the  implementat ion  of  operations  research  solu¬ 
tions.  The  study  sample  was  composed  of  107  OR  p;rojects  proposed  for  im¬ 
plementation;  29  of  the  projects  were  not  imp  lemented  by  the  user. 

The  results  of  the  survey  are  {^resented  in  Table  2.  There  is  a  posi¬ 
tive  relationship  between  implementation  and  (1)  the  user's  collaboration  i 
defining  the  problem,  (2)  problem  characteristics,  and  (3)  the  value  the 
user  pilaces  on  the  proposed  OR  solution.  All  three  factors  require  contin¬ 
ual  interaction  between  the  user  and  the  OR  modeler  throughout,  the  process 
of  model  deve  lopjment . 


18 


Table  2 


Summary  of  Survey  Results:  Relationships  between  Variables  and  Proposal 
Implementation  (from  Lonnstedt,  1975) 


Variable  group 

Variable 

Chi 

square 

Significance 

(p) 

Influence  of 

nonresponses 
on  conclusion 

Col laboration 

User's  participation 

19.1 

<  .001 

May  influence 

Initiator  of  project 

16.7 

<  .001 

May  influence 

Characteristic 

Problem  limitation 

52.5 

■  .0001 

No  influence 

of  problem 

Quantif iability  of 
variables 

24  .4 

<  .0001 

No  influence 

Availability  of  data 

33.0 

-  .0001 

No  influence 

Proposal  value 
and  cost 

Value  of  resultant 
solution 

34.1 

<  .0001 

May  influence 

Internal  charging 

0.85 

<  .4 

May  influence 

Shycon  (1977)  conducted  two  surveys  of  large  OR  projects  varying  in  their 
degree  of  successful  implementation.  Both  surveys  categorized  each  OR  project 
team's  organization  as  one  of  the  following  three  types:  (1)  the  wholly 
management  science  team  consisting  entirely  of  management  science  personnel, 
with  minimal  interaction  with  others;  (2)  the  management  science  team  with 
marginal  communication  to  management,  largely  at  the  middle  management  level, 
through  frequent  reporting;  and  (3)  the  interparticipative  management  science, 
management  team,  which  involves  a  working  partnership  of  members  of  the  man¬ 
agement  science  group'  and  middle  and  upper  management  representing  both  line 
and  staff  functions.  Under  this  classification  system,  the  project  team  for 
implementing  R-SCREEN  was  a  wholly  management  science  team.  In  contrast, 
the  team  for  MCCRESSA  was  an  interparticipative  management  science/management 
team. 


Table  3  presents  the  results  of  Shycon's  (1977)  surveys.  The  wholly 
management  science  team  achieved  the  lowest  degree  of  implementation  success. 
The  MS  team  with  communication  and  the  interparticip^ative  team  consistently 
achieved  high  levels  of  implementation  success.  Neither  of  the  latter  two 
organization  types,  however,  showed  any  distinct  advantage  over  the  other. 
Given  the  greater  cost  of  the  interparticipative  team,  the  results  suggest 
that  the  MS  team  with  communication  is  the  most  cost-effective  project  team 
organization.  Nevertheless,  the  results  support  p>ost  hoc  the  actual  outcomes 
for  R-SCREEN  and  MCCRESSA  regarding  implementation  success. 
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Table  3 


Project  Team  Organization  and  Project  Success 


(from  Shycon, 


1977) 


Type  of  project 


Approx  . 

project  k 

Type  of  firm  costd  Degree  of  success. 


1971 


1 .  Wholly  management  science  team 

Determination  of  regional  distribution  requirements 
Sales  forecasting  and  inventory  planning  system 
Sales  forecasting  and  inventory  planning  system 

Determination  of  service  call  response  strategy 
and  facilities  required 

Determination  of  service  call  response  strategy 
and  facilities  required 

2 .  MS  team  with  marginal  contnuni  cat  ion 
Design  of  national  distribution  system 

Design  of  total  logistics  system,  manufacturing 
plants,  and  distribution 
Service  facility  requirements 

Integrated  distribution  requirements  for  diverse 
divisions 

3 .  Fully  intert articipative  MS  team 

Total  management  planning  program:  pro.  urem-i.t  , 
inventory,  scheduling,  distribution,  salt- 
forecasting,  and  marketing  planning 
Design  of  national  distribution  system 
Design  of  national  distribution  system 

Basic  simulation  of  company  operations  for  man¬ 
agement  policy  testing 
Evaluation  system  for  R£D  projects  and  tool 
program  for  rank  order  and  funding 
Corporate  strategy  model  for  decisions  in  mar¬ 
keting,  manufacturing,  and  capital  investment 
Design  of  total  logistics  system,  manufacturing 
plants,  and  distribution 


Regional  food  distribution 
National  tool  manufacturer 
Division  of  major  druq 
manufacturer 
Regional  public  utility 

Regional  public  utility 
(different  from  above) 


National  tood  processor 
National  food  processor 

Heavy  machinery  manufacturei 
Major  drug  and  toiletries 
manufacturer 


M.i'.or  r.at.i.a.  meat  jacket 

Natiorul  tood  processor 
Major  instrument  and  sup¬ 
plier  manufacturer 
Major  pharmaceutical 
manufacturer 
Major  synthetic  fiber 

Major  integrated  paper 
products  manufacturer 
National  industrial  prod¬ 
ucts  plastics  manufacturer 


3(. 

Partial 

(ib 

Partial,  long-term 

3b 

Little  immediate, 

some  longer-term 

30 

Complete 

32 

Partial 

14u 

9r 

Com]  lete 
Complete 

55 

54 

Complete 

Minor  benefits 

■ 

C  *mj  1 1- 1 « 

1. 

9e 

Complete 

Complete 

104 

Complete 

1*.0 

Partial 

250 

Complete 

75 

Complete 
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1 .  Wholly  management  science  team 
Determination  of  national  distribution  and 

transportation  strategy 
Multiplant  manufacturing  operational  strategy 

2 .  MS  team  with  marginal  communication 
Research  and  development  of  invent*  ry  jian::nr 

system 

Development  of  marketing  strati.-;*/  with  at  t  r ».  i .  i : .  • 
distribution  requirements 
Queuing  simulation  model,  flow  sho; 

Evaluation  of  marketing  channels  and  design  of 
order  entry 

3 .  Fully  i nterparticipat i vc  MS  team 
Simulation  and  design  of  national  distribution 

system 

Evaluation  of  customer  service  requirements  and 
design  of  national  distribution  system 
Design  of  integrated  strategy  planning  model 

Development  of  marketing  channels  and  supply 
strategy 

Development  of  specifications  for  inventory 
management  system  and  implementation 
Design  of  national  distribution  system 


Consumer 

large  instruments 

4  2 

Li ttle 

Building 

products  mar.ufac  turcr 

?7 

Little 

Sport ina 

<H<  Is  manufacturer 

1  1 

Corj  Jet  I* 

Aut  •  >nx.L  i  1 

manuf ac 

e  alter  market 

» ill  e  • 

Conj  lit. 

Principal  furr.ituie 
manufacturer 

Part  i  .>  i 

Food  manuf acturer 

B0 

Complete 

Chemical  processor 

Part iat 

National  food  processor 

i  i. 

Com;  let  v 

International  extractive  and 

10b 

Partial 

and  fabrication  company 

High  technology  industrial 

94 

Complete 

product  con^iany 

Principal  food  processoi 

2b5 

Oom^lete 

Industrial  products  hard 

85 

Complete 

good  manufacturer 


includes  all  services,  internal  personnel,  and  external  expenses  incurred.  All  figures  for  1971  adjusted  to 
1971  dollars.  All  figures  for  1977  adjusted  to  1977  dollars. 

bDegree  of  success  is  necessarily  partly  subjective;  however,  measurable  criteria  are  as  fnllws:  little-- less 
than  25%  implemented,  benefits  did  not  justify  cost  of  study;  par t ial--25-60%  implemented,  identifiable  benefits 
yield  return  on  project  investment  (ROPI)  less  than  100%  per  year;  compl et e--great e t  than  U'%  imj  lement at  ion , 
ROP1  greater  than  100%  .per  year. 
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Why  User  Involvement  Is  Essential 


User  involvement  throuqhout  the  development  of  decision-analytic  aids 
is  essential  to  make  users  comfortable  with  a  decision-analytic  approach  t> . 
decision  making.  User  involvement  is  also  essential  if  the  analysts  are  t< 
learn  enough  about  the  users'  goals  and  working  environment  to  tailor  t:.e 
decision-analytic  aid  to  the  users'  jjersonal  needs  and  organizational  c<  r.text  . 

Decision  makers  are  not  decision  analysts.  Although  they  may  ide:.ti:y 
important  factors  for  the  decision  at  hand ,  they  will  seldom  build  a  dec : on 
tree  or  multi -at tribute  hierarchy.  Nor  will  they  typically  quantify  the 
probability  of  uncertain  events  or  the  relative  importance  e 1  attributes 
over  their  range  of  variation  tor  the  set  of  al ternat i ves ,  or  calculate  ex¬ 
pected  utilities  to  determine  the  pretexted  action.  Since  formal  dt.v  isi< 
analysis  is  not  their  standard  mode  of  decision  making ,  decision  maker;  nee-: 
to  learn  basic  decision-analytic  concet  ts  and  feel  comfortable  5 roviditc:  tin 
aid's  required  irq  uts  and  interpreting  its  outj  uts  before  they  will  use  it 
routinely.  Interaction  between  the  analyst  and  the  decision  maker'  is  essen¬ 
tial  to  this  learning  process  and,  m  -re  generally,  to  devolej  i no  the  cinti- 
dertce  and  commitment  necessary  for  im|  leirent  im:  a  di  fierent  aj  j  roach  to  de¬ 
cision  making. 

The  interaction  must  be  a  two-way  ;  1 ocess ,  however,  for  the  analysts 
must  understand  the  organization's  broader  goals,  working  environment ,  and 
available  resources  in  order  to  develoj  an  effective  decision-analytic  aid 
with  stand-alone  caj  abilities.  For  example,  the  analysts  must  understand 
how  the  decision  makers  want  to  use  the  aid  m  order  to  design  its  output 
so  that  it  most  effectively  meets  the  decision  makers’  goals.  The  analysts 
need  to  understand  the  different  tasks  required  to  achieve  these  goals,  the 
different  types  of  people  who  will  j erform  these  tasks,  and  the  factors 
that  facilitate  or  limit  task  accomplishment  in  order  to  design  the  aid  so 
that  it  not  only  fits  into,  but  improves,  the  working  environment.  And  the 
analysts  need  to  know  the  expected  fiscal  resources  available  for  q  ota- 
tionally  utilizing  the  aid  so  that  it  is  designed  cost-effectively.  Involv¬ 
ing  the  user  throughout  the  development  process  increases  the  p'robob 1 1 1 t y 
that  analysts  will  obtain  such  information  about  the  organization. 

Knowledge  about  the  working  environment  within  which  an  aid  will  be 
used  permits  analysts  to  tailor  the  aid  to  the  users'  personal  needs  in 
their  organizational  context.  The  need  for  such  tailoring  has  become  in¬ 
creasingly  documented  in  the  OR/MS  literature.  For  example,  in  an  analysis 
of  implementation  of  risk  analysis  methods.  Carter  (1972)  found  that  unsuc¬ 
cessful  efforts  tended  to  have  analyses  performed  by  central  staffs  respon¬ 
sible  to  corporate  rather  than  divisional  managers.  Division  managers  per¬ 
ceived  a  divided  loyalty  within  the  management  science  staff;  this  perception 
resulted  in  a  breakdown  in  trust  and  cooperation.  Wolek  (1975)  cites  a 
case  in  which  a  rational  system  for  selecting  R&D  projects  was  formally 
adopted  but  not  used  because  the  technique  conflicted  with  the  highly  per¬ 
sonal  leadership  style  of  the  company's  president.  And  the  R-SCREEN  exam; le 
illustrates  that  a  decision-analytic  aid  that  is  beneficial  will  not  neces¬ 
sarily  be  adopted;  the  aid  also  has  to  score  well  on  behavioral 
characteristics . 
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A  decision-analytic  aid  will  change  the  users'  working  environment,  ioi 
it  will  alter  the  organization's  decision-making  process.  This  change  will 
occur  whether,  according  to  Von  Winterfeldt 's  (1979)  classification,  the  aid 
is  (1)  a  highly  specific  aid  with  a  previously  determined  structure  and 
stored  data  (e.q.,  weights)  like  MCCRESSA,  (2)  a  multipurpose  aid  with  no 
substantive  structure  and  stored  data,  or  (3)  an  aid  like  R-SCREEN  that  com¬ 
bines  features  of  both  extremes.  However,  the  extent  of  the  change  and  its 
effect  on  the  interpersonal  relations  and  functions  of  different  people 
within  the  organization  will  vary .  In  addition,  it  may  be  necessary  to  hire 
(or  train  internally)  skilled  decision  analysts  to  use  effectively  multi¬ 
purpose  aids  with  no  previously  determined  structure.  Such  aids  require 
more  operational  support  than  either  of  the  other  two  types  because  they  are 
not  used  for  repetitive  decisions.  In  contrast,  structured  aids  used  for 
repetitive  decision  making,  like  MCCRESSA,  require  little  operational  su:  - 
port  because  their  analytical  structure  is  designed  primarily  to  implement 
an  existing  on-going  process  more  effectively.  Nevertheless,  the  smoother 
the  expected  and  actual  transition  to  a  new  decision-making  approach  and 
working  environment,  the  greater  the  probability  of  successful  aid  imple¬ 
mentation.  User  involvement  throughout  the  process  of  aid  development  and 
subsequent  implementation ,  that  is,  "from  initial  ] lanning  and  feasibility 
testing  through  installation  and  evaluation"  (Ginzberg,  1978,  p.  59) ,  can 
ease  this  transition  greatly. 

In  closing,  it  is  interesting  to  note  that  the  need  for  user  involve¬ 
ment  in  aid  development  may  not  surprise  many  decision  analysts  and  re¬ 
searchers.  What  may  surprise  them  is  the  empirical  support  for  this  need; 
there  are  many  cases  in  the  OR.  MS  literature,  and  now  some  m  the  decision- 
analytic  literature,  in  which  users  have  not  been  involved  in  aid  develoj - 
ment.  Two  possible  reasons  for  tins  state  of  affairs  come  rapidly  to  mind. 
First,  many  decision  analysts  and  operations  researchers  have  not  realized 
the  importance  of  user  involvement  for  successful  implementation .  It  is  oin- 
thing  to  give  the  concept  of  user  involvement  lip-service  and  quite  another 
to  consider  it  the  principal  factor  in  successful  implementation.  Second, 
ensuring  user  involvement  is  a  difficult  task.  Users  often  fail  to  aitreci- 
ate  the  importance  of  their  involvement  and,  as  a  result,  consider  aid  de¬ 
velopment  to  be  solely  the  tob  of  the  analyst  instead  of  a  two-way  lr.t.ei. ac¬ 
tion.  The  primary  usefulness  of  this  section  may  well  lie  in  alert  me 
analysts  to  the  necessity  of  makinq  users  realize  the  importance  el  their 
involvement  in  aid  development. 


DECISION  AID  EVALUATION 

The  preceding  sections  have  been  concerned  with  the  problem  ot  design¬ 
ing  effective  decision  aids,  but  the  present  section  turns  to  the  problem 
of  evaluating  decision  aids.  Although  design  and  evaluation  are  highly 
interrelated,  they  are  approached  from  different  perspectives  and  therefon- 
involve  somewhat  different  difficulties.  Design  often  begins  in  the  ab¬ 
sence  of  an  implemented  system  and  must  determine  how  to  incorporate  a  vari¬ 
ety  of  capabilities.  Evaluation  begins  with  a  partial ly  or  completely  im¬ 
plemented  system  and  must  determine  whether  the  system  does  what  it  is  meant 
to  do.  Optimally,  the  two  activities  are  performed  iteratively,  vi t h  the 


answers  to  design  questions  posing  evaluation  questions,  and  the  answers  to 
evaluation  questions  posing  design  problems.  Nonetheless,  they  are  quite 
distinct  activities. 


Three  major  types  of  questions  relate  to  the  evaluation  of  decision 
aids;  these  questions  correspond  to  the  three  types  of  interfaces  identified 
in  the  introduction  to  this  report  (Figure  1).  First,  an  evaluation  may  at¬ 
tempt  to  answer  questions  about  the  aid's  compatibility  with  its  immediate 
users.  Such  questions  are  concerned  with  the  human  factors  of  the  aid;  for 
example:  Are  its  displays  effective?  Is  it  tedious  to  use?  In  addition, 

the  decision  aid/user  interface  is  the  point  at  which  questions  about  the 
comprehensibility  of  an  aid's  underlying  model  are  addressed. 

A  second  type  of  evaluation  occurs  at  the  interface  between  the  user 
(and  the  decision  aid)  and  the  remainder  of  the  decision-making  organization 
This  interface  poses  questions  about  the  collectibility  of  the  aid's  re¬ 
quired  inputs  and  the  communicability  of  its  outputs.  An  aid  that  is  com¬ 
prehensible  only  to  its  immediate  user  is  likely  to  be  useless.  The 
decision-making  approach  used  by  the  aid  must  be  integrated  into  the 
larger  decision-making  organization. 

Finally,  a  third  type  of  evaluation  is  appropriate  for  the  interface 
between  the  decision-makir.g  organization  and  its  environment.  At  this  point 
the  ultimate  question  of  the  aid's  effectiveness  comes  into  play;  namely, 
has  the  aid  improved  the  organization's  output  or  perf ormance .  Similarly, 
there  are  questions  about  the  range  of  environments  or  problem  areas  over 
which  the  aid  provides  improved  organization  performance. 

These  three  types  of  interfaces — decision  aid/user,  user/organization , 
organization, 'environment — are  by  no  means  1 ndependent .  In  fact,  they  are 
nested:  User/organization  effectiveness  is  necessarily  influenced  by  aid, 

user  effectiveness,  and  organization/environment  effectiveness  is  necessaril 
influenced  by  the  effectiveness  of  the  other  two  interfaces.  Nevertheless, 
the  three  types  of  interfaces  do  have  different  implications  for  evaluation, 
which  justifies  their  use  as  a  framework  for  discussing  aid  evaluation. 

The  subsections  that  follow  use  this  framework  to  examine  several  as¬ 
pects  of  evaluation.  The  first  subsection  considers  the  problem  of  identi¬ 
fying  measures  of  effectiveness.  The  second  addresses  the  selection  of  a 
setting  for  conducting  the  evaluation.  The  third  discusses  the  selection 
of  a  method  for  data  collection.  And  the  fourth  discusses  the  question  of 
what  is  being  compared  in  the  evaluation. 


Measures  of  Effectiveness 


If  an  evaluation  is  to  be  effective,  the  evaluator  must  decide  in  ad¬ 
vance  what  is  to  be  examined.  This  is  done  by  identifying  one  or  more 
measures  of  effectiveness  that  are  designed  to  answer  the  evaluator's  ques¬ 
tions.  Ideally,  these  measures  of  effectiveness  are  objectively  measurable 
and  quantitative  variables  that  will  describe  the  effectiveness  of  the  aid. 
In  the  present  case,  however,  the  term  measure  of  effectiveness  will  also 
include  subjectively  measurable  variables  and  variables  that  result  in 


qualitative  rather  than  quantitative  descriptions .  The  only  restrictions 
are  that  the  variable  must  be  measurable  and  that  it  should  be  expected  to 
correlate  (positively  or  neqatively)  with  the  effectiveness  or  efficiency 
of  the  aid. 

Althouqh  it  would  be  impossible  to  list  the  measures  that  are  appro;  ri- 
ate  to  all  evaluations,  another  approach  may  provide  some  insight.  The  j  re¬ 
cess  by  which  an  aid  is  used  can  be  viewed  as  a  series  of  stages  that  proceed 
from  initial  data  collection  to  decision  implementation  (see  Figure  2). 

Under  this  assumption,  anything  that  improves  the  effectiveness  or  efficiency 
of  one  stage  should  improve  the  overall  effectiveness.  Thus,  the  discussion 
of  measures  of  effectiveness  can  be  simplified  somewhat  by  discussing  how  the 
aid  can  affect  each  stage. 

Data  Collection.  Data  collection  is  the  stage  during  which  the  decision¬ 
making  organization  extracts  information  from  its  environment.  In  a  military 
context,  tins  is  the  domain  of  intelligence.  In  the  government,  it  is  the 
domain  of,  for  example,  the  Census  Bureau  or  the  Bureau  of  Labor  Statistics. 

In  the  present  context,  it.  is  part  of  the  organization/environment  interface. 

At  first  glance,  this  stage  might  seem  remote  from  the  aid  and  therefore 
not  subject  to  its  influence.  Nothing  could  be  further  from  tiie  truth.  The 
introduction  of  an  aid  has  the  potential  to  qreatly  improve  data  collection 
by  demanding  a  better  organized  and  better  quantified  measurement  of  the  en¬ 
vironment.  In  addition,  the  aid  could  possibly  directly  interface  with  the 
sensing  equipment,  which  could,  in  some  instances,  eliminate  errors  that 
might  otherwise  be  introduced  as  the  data  are  transferred  through  the  organi¬ 
zation  to  the  user  and  into  the  aid. 

A  decision  aid  could,  however,  have  a  negative  impact  on  the  data  col¬ 
lection  stage.  If  the  model  underlying  the  aid  were  either  inaccurate  or 
unintuitive,  it  could  compel  the  collection  efforts  to  be  misdirected.  Also, 
an  increased  data  collection  effort  might  be  required,  thereby  increasing 
costs . 

Data  Interpretation.  The  data  interpretation  staqe  is  the  stage  during 
which  the  members  of  the  organization  transform  and  otherwise  interpret  the 
information  about  the  environment.  This  process  may  follow  strict  procedures 
or  it  may  involve  subjective  judgments  concerning  the  implications  of  the 
data.  Although  this  process  may  take  place  throughout  the  organization,  it 
is  considered  here  as  part  of  the  user/orqanization  interface,  since  this  is 
the  point  at  which  the  final  judgments  must  be  made. 

An  aid  could  improve  the  data  interpretation  stage  by  improving  the  or¬ 
ganization's  ability  to  focus  on  critical  information.  By  disaggregating  a 
problem  into  meaningful  and  manageable  subproblems,  the  aid  may  indicate  how 
the  data  should  be  organized,  how  the  data  should  be  transformed,  or  what 
types  of  judgments  will  be  required.  Also,  the  aid  may  compel  a  more  care¬ 
ful  identification  of  options  than  is  normally  undertaken. 

In  contrast  to  these  benefits,  an  aid  may  introduce  substantial  costs 
to  the  data  interpretation  stage.  The  aid  might  require  specially  trained 
personnel;  it  might  increase  the  overall  workload  by  demanding  inputs  that 
would  not  otherwise  be  collected;  and  it  might  create  a  strain  by  requiring 
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orqunization  members  to  think  in  a  manner  that  is  neither  natural  nor 
intuitive. 

Data  Entry.  The  data  entry  stage  is  the  period  dunnq  which  the  user 
provides  the  aid  with  its  required  inputs.  This  staqe  is  part  of  the  aid 
user  interface. 

Data  entry  is  a  necessary  and  frequently  arduous  aspect  of  decision  aid 
usage.  For  this  stage  an  aid  will  be  evaluated  on  the  ease  with  which  the 
data  entry  can  be  performed.  The  aid  should  make  data  entry  as  rapid  as  pos¬ 
sible  and  permit  a  wide  range  of  editing  options.  The  aid  should  enable  un¬ 
trained  personnel  to  perform  data  entry,  and  it  should  pjrovide  easily  under¬ 
stood  prompts  to  help  the  user  accomplish  this  task.  Finally,  the  decision 
aid  should  place  minimal  psychological  discomfort  on  the  user.  These  points 
are  discussed  in  detail  earlier  in  this  report. 

Decision  Aid  Output.  The  output  stage  is  the  period  during  which  an  am 
provides  a  user  with  its  results.  This  stage  is  clearly  part  of  the  aid, 
user  interface.  The  user  for  this  stage,  however,  need  not  be  the  same  in¬ 
dividual  as  the  user  in  the  data  entry  stage. 

During  the  data  output  stage  the  aid  has  its  most  obvious  and  immediate 
opportunity  to  be  of  value  to  the  decision-making  organization.  To  accompli? 
this  the  aid  must  provide  rapid,  thorough,  and  effective  interaction  with  the 
user.  Sensitivity  analyses  are  critical,  because  they  inform  the  user  of  the 
aspects  of  the  decision  that  require  further  inspection.  Moreover,  the  dis¬ 
plays  must  be  both  accurate  and  interpretable  so  that  the  user  will  readily 
understand  the  underlying  model  and  why  it  has  provided  the  displayed  results 

These  benefits  of  an  aid  will  not  be  without  cost,  including  the  cost 
of  the  equipment  itself.  Usage  may,  at  this  stage  more  than  any  other,  re¬ 
quire  a  specially  trained  user.  This  is  the  point  at  which  a  careful  under¬ 
standing  of  the  underlying  model  and  the  options  available  for  its  explora¬ 
tion  can  pay  off.  The  job  of  conveying  the  model's  results  to  the  organizati 
will  fall  upon  this  user. 

Decision  Aid  Output  Interpretation.  The  output  interpretation  stage  is 
the  period  during  which  the  implications  of  the  aid’s  analysis  are  conveyed 
to  the  organization  and  a  decision  is  made.  Althouqh  this  stage  involves 
many  actors  in  the  organization  and  may  only  briefly  involve  the  user,  it  is 
considered,  nonetheless,  to  be  part  of  the  user/organization  interface. 

The  decision  aid's  influence  on  this  stage  will  depend  on  the  aid's 
ability  to  structure  and  organize  the  problem  to  which  it  was  apiplied.  If 
the  model  is  conceptually  complete,  coherent,  and  rational  from  the  point 
of  view  of  the  decision  makers  within  the  organization,  then  its  results 
have  a  chance  of  acceptance.  Especially  important  are  the  communicabili ty 
and  justifiability  of  the  results'  implications.  If,  however,  anyone  in 
the  chain  of  communication  leading  to  the  decision  makers  or  the  decision 
makers  themselves  feel  annoyed  at  or  uncertain  about  the  results,  then  these 
results  are  likely  to  be  ignored  and  possibly  suppressed  from  that  point  on. 
One  way  to  minimize  this  problem  is  to  have  users  (both  hands-on  users  and 
decision  makers)  involved  in  designing  the  decision  aid.  This  develops  the 
understanding  and  commitment  necessary  for  implementing  a  different 


decision-making  approach,  and  the  characteristics  of  the  decision  aid  can  be 
tailored  to  the  needs  of  users  within  their  organization. 

Decision  Implementation .  The  decision  implementation  stage  is  the  pe¬ 
riod  during  which  a  decision  is  translated  into  some  action  on  the  part  of 
the  organization.  In  the  case  of  operational  decisions,  this  stage  is  part 
of  the  organizat ion/ envi ronment  interface.  Internal  development  decisions, 
those  directly  affecting  the  organization,  are  not  actually  excluded  from  the 
framework,  but  there  is  no  interface  required  with  the  environment.  Instead, 
the  organization  should  be  depicted  as  feeding  back  cn  itself  in  a  self- 
regulatory  fashion. 

For  this  stage  the  most  fundamental  question  is  whether  the  aid  has  led 
to  a  sound  decision.  If  the  decision's  implications  are  correct  and  they  are 
not  ignored,  then  the  aid  has  provided  its  major  benefit.  The  aid  can,  in 
addition,  provide  insight  into  how  the  decision  should  be  implemented  and 
what  is  likely  to  occur  following  implementation.  Even  at  this  stage,  how¬ 
ever,  a  correct  decision  could  be  undermined,  if  it  is  both  counterintuitive 
and  unjustifiable.  Thus,  the  communicability  and  comprehensibility  of  the 
aid  must  carry  through  even  to  this  late  stage  of  the  process. 

Summary  of  Potential  Measures  of  Effectiveness.  To  provide  a  summary  of 
some  potential  measures,  Figure  3  reiterates  many  of  the  issues  raised  in  the 
preceding  discussion.  In  this  figure,  the  issues  are  organized  into  a  hier¬ 
archy,  and  it  is  assumed  that  each  terminal  node  could  be  translated  into  a 
measure.  This  representation  is  not  meant  to  advocate  any  particular  set  c.  i 
measures  or  any  particular  approach  to  evaluation.  It  is  simply  a  summary 
of  a  number  of  measures  that  may  be  relevant  to  any  specific  decision  aid 
evaluation . 

Organizational  Impact.  Although  it  is  implicit  in  the  comments  of  the 
preceding  discussion,  one  additional  point  deserves  mention.  The  intrcuuctic 
of  an  aid  into  a  complex  organization  is  unlikely  to  be  accomplished  without 
changes  in  the  organization.  A  decision  aid  is  not  like  a  new  stereo  com¬ 
ponent  that  can  simply  plug  into  the  old  system  as  a  replacement  for  some 
older  component.  For  one  thing,  the  aid  will  be  unlike  any  existing  com;  i. 
that  is,  individual,  of  the  organization.  It  cannot  truly  replace  a  perse:., 
because  it  cannot  do  all  that  a  person  does.  This  fact  implies  that  the  in¬ 
troduction  of  the  aid  will  necessarily  redefine  certain  roles  within  the 
organization . 

The  impact  of  a  decision  aid  on  an  organization  may  in  fact  be  swot; in:. 
To  use  an  aid,  new  channels  of  communication  may  be  required  and  new  area:  >  ; 
authority  may  need  to  be  defined.  Such  changes  could  be  minimized  by  a  c.,i «— 
ful  design  effort  prior  to  development  of  the  aid.  Nonetheless,  some  orcan- 
zational  change  is  likely  to  be  necessary.  In  the;  event  that  the  chaiioe  1 .. 
too  great,  one  can  expiect  the  aid  to  lie  idle.  However,  if  the  orgum  z.a  t  i  ■  :.u 
changes  are  slight  or  at  least  carefully  planned,  the  aid  will  have  a  chan. e 
to  take  hold  and  contribute  to  the  effectiveness  of  the  decision-mak  1  n  : 
organization . 
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Figure  1.  Summary  of  potential  mcasuies  of  e  1  1  ect  i  vonos 


Settings  rot  Decision  Aid  E valuations 

Before  the  measures  of  effectiveness  (or  a  decision  aid  can  be  col lectv 
and  analyzed,  it  is  necessary  to  construct  a  setting  in  which  the  aid  can  to- 
operated.  The  setting  might  simply  be  a  laboratory'  experiment  with  a  rock 
problem,  or  it  might  be  a  full  field  trial.  Such  settings  differ  in  terms  c 
their  fidelity,  or  similarity,  to  the  expected  operational  setting,  the  ar.ou 
of  experimental  control  that  they  provide,  and  their  costs.  Thus,  the  cnoit 
of  a  setting  can  be  a  difficult  one. 

Figure  4  depicts  the  situation  that  prevails  when  one  attempts  to  con¬ 
duct  an  assessment  of  effectiveness.  The  first  part  of  the  figure,  labeled 
Target  Setting,  represents  the  expected  operational  setting  for  the  aid.  ■  ! 
course,  tins  setting  will  not  be  available  for  evaluation  purposes  unless  t.n 
aid  is  actually  deployed.  In  lieu  of  the  target  setting,  it  is  therefore 
necessary  to  construct  a  test  setting  within  which  the  evaluation  can  t  meet 

One  of  the  most  fundamental  dimensions  over  which  test  settings  can  Vji 
is  their  degree  of  fidelity  to  the  target  setting.  The  simulated  environ¬ 
ment  ,  the  simulated  organization,  and  even  the  simulated  user  can  range  be¬ 
tween  being  only  superficially  accurate  to  being  accurate  in  great  detail. 

By  itself,  high  fidelity  is,  of  course,  desirable  in  any  evaluation  setting, 
but  it  is  expensive.  In  addition  to  increased  dollai  costs  and  evaluation 
time,  fidelity  introduces  a  cost  in  terms  of  loss  of  exj erimenter  centred. 
This  means ,  on  the  one  hand,  that  it  may  be  mcrea:-  inuly  difficult  tv  obtain 
the  desired  measures  and,  on  the  other  hand ,  that  these  measures  will  be  in¬ 
creasingly  susceptible  to  influences  that  are  extraneous  to  the  evaluation 
context.  Even  if  one  is  successful  in  eliminating  ext  raneous  influences 
from  the  evaluation,  there  will  be  increased  difficulty  it:  sj  ecifyinu  am: 
controlling  causal  relationships  in  a  hiqh-f idc li ty  Setting  .  Thus ,  a  trade¬ 
off  is  established  between  fidelity  and  costs  such  that  it  is  desirable-  tt 
simulate  only  as  much  of  the  target  setting  as  is  necessary  to  support  a 
particular  inference. 

Tins  concept  of  fidelity  and  the  trade-offs  it  implies  car.  be  further 
examined  by  considering  four  settings  defined  by  whether  they  involve  high 
or  low  fidelity  for  the  organization  and  high  or  low  fidelity  for  the  en¬ 
vironment.  The  additional  settings  provided  by  low  fidelity  for  the  use! 
are  not  examined  because  a  qualified  user  is  required  for  any  evaluate 

Of  the  four  settings,  the  setting  with  a  low-fidelity  environment  and 
low-fidelity  organization  is  the  most  austere.  Such  a  setting  is  well  suiti 
to  decision  aid  design  questions  concerning  user  compatibility.  These  ques¬ 
tions  are  primarily  concerned  with  the  aid/user  interface  and  therefore  need, 
not  concern  the  organization  or  even  the  true  environment.  Since  so  little 
simulation  effort  is  required,  this  setting  can  be  implemented  in  a  care! all 
controlled  laboratory  experiment. 

The  setting  with  a  high-fidelity  environment  and  low-fidelity  organiza¬ 
tion  can  also  be  implemented  in  the  laboratory,  but  it  serves  a  different 
purpose.  This  setting  focuses  on  providing  the  user  with  realistic  data 
about  the  environment,  realistic  options,  and  realistic  scenarios,  but  the 
organization  through  which  the  user  would  interact  with  this  environment  is 
only  superficially  implemented .  Thus,  it  is  possible  to  investigate  the 
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coherence  and  completeness  of  the  model  underlying  the  aid  without  going  tin- 
additional  step  and  ascertaining  whether  the  aid  will  improve  organization 
performance.  Such  an  analysis  really  evaluates  only  questions  related  to 
the  aid.  user  interface. 

The  setting  with  a  low-fidelity  environment  and  hig'n-f idel ity  organi¬ 
zation  provides  the  means  to  answer  questions  about  the  user/organizaticn 
interface.  In  this  setting,  little  concern  is  devoted  to  constructing  real¬ 
istic  problems,  and  a  great  deal  of  concern  is  devoted  to  simulating  the 
lines  of  communication  and  authority  within  the  organization.  Since  a  simu¬ 
lated  organization  is  outside  the  scoj  e  of  most  laboratories,  this  type  cf 
setting  is  better  thought  of  as  a  gaming  simulation.  Although  the  departuro 
from  the  laboratory  is  necessary  to  evaluate  the  user/organi zation  inter¬ 
face,  this  setting  implies  decreasing  control  and  increasing  costs. 

Finally,  the  setting  with  a  hiah-f idel ity  environment  and  high-fidelity 
organization  is  the  most  accurate,  virtually  requiring  a  field  test  with  a 
realistic  and  well-implemented  problem  scenario.  This  accuracy  is  obtained 
at  a  high  cost,  but  it  is  necessary  to  fully  answer  questions  about  decision 
aid  effects  on  organization  performance.  Since  these  questions  about  the 
organization/ environment  interface  are  the  ultimate  questions  concerning  aid 
effectiveness,  field  tests  of  this  sort  are  a  highly  desirable  precursor  tc 
aid  deployment. 

Table  4  summarizes  the  comments  cf  the  preceding  paragraphs.  Clearly, 
the  choice  cf  an  evaluation  setting  interacts  with  the  type  of  question  that 
one  hopes  to  answer.  In  light  of  these  fidelity/cost  trade-offs,  the  follow¬ 
ing  approach  to  evaluation  seems  justified. 

If  all  types  of  questions  are  important,  investigate  them  in  the  follow¬ 
ing  order:  questions  of  user  compatibility,  questions  of  aid  coherence  and 
completeness,  questions  of  aid  compatibility  with  the  organization,  and  ques¬ 
tions  of  aid  effect  on  organization  performance.  Although  the  order  of  the 
first  two  evaluations  may  change,  cost  considerations  are  likely  to  compel 
the  remainder  of  this  evaluation  strategy,  since  it  will  be  desirable  tc  have- 
suffered  the  least  costs  in  the  event  that  any  one  evaluation  provides  a  nega¬ 
tive  result. 


Methods  for  Collecting  Measures  of  Effectiveness 

There  are  three  major  methods  for  obtaining  measures  of  effectiveness: 
objective  measurement,  subjective  judgment,  and  expert  observation.  The 
first  of  these  is  the  most  familiar  and  is  most  associated  with  experimenta¬ 
tion  and  the  scientific  method.  The  second  technique,  subjective  judoment, 
involves  requiring  users  or  other  participants  in  the  experiment  to  score 
their  experiences,  usually  via  a  questionnaire  following  the  experiment. 

The  final  technique,  expert  observation,  also  involves  subjective  judgments, 
only  on  the  part  of  nonparticipat inq  observers  of  the  experiment.  Although 
there  is  a  prevailing  prejudice  in  favor  of  objective  measurement,  all  of 
these  methods  can  be  valid  provided  they  are  properly  employed. 

In  decision  aid  evaluation,  objective  measurements  are  likely  to  con¬ 
sist  of  sp)eed  and  frequency  measures.  It  will  be  important  to  know  how  long 
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some  process  requires  or  how  frequently  errors  occur.  Table  5  suggests  ways 
in  which  such  objective  measurements  could  be  used  to  evaluate  the  aid/user, 
user/oryamzation ,  and  organization/environment  interfaces. 

No  less  important  than  objective  measures  are  the  assessments  of  a  par¬ 
ticipant's  satisfaction,  complaints,  or  other  judgments  about  an  aid's  ef¬ 
fectiveness.  Not  only  are  these  judgments  easier  to  collect  than  objective 
measures,  but  they  represent  a  class  of  data  that  is  a  critical  determinant 
of  an  aid's  ultimate  acceptability.  An  aid  that  is  objectively  effective, 
yet  subjectively  unacceptable,  is  still  unacceptable,  since  its  chances  for 
effective  deployment  are  low.  Thus,  subjective  judgments  should  not  be  ig¬ 
nored.  Table  5  suggests  several  measures  of  effectiveness  that  measure  sub¬ 
jective  ludgments  at  each  of  the  three  interfaces. 

The  final  method  of  evaluation,  expert  observation,  differs  from  sub¬ 
jective  judgment  in  that  the  raters  are  not  participants  in  the  experiment 
but  outside  observers.  Judgments  of  this  sort  can  be  critical  for  answer:;,  : 
questions  about  the  completeness  or  soundness  of  a  decision,  because  the 
correct  decision  is  unlikely  to  be  known.  In  the  absence  of  any  objective- 
definition  of  correctness  or  accuracy,  expert  judgments  must  suffice.  In 
this  capacity  the  experts  play  the  same  role  that  a  coach  or  trainer  plays. 
Experts  die  deemed  correct  by  virtue  of  their  greater  experience.  Tabic-  L- 
suggests  some  w.  s  that  expert  observations  can  assist  an  evaluation  effort. 

I  summary,  each  of  the  three  methods  of  collecting  measures  of  effec¬ 
tiveness  plays  an  imjortant  role  at  all  three  interfaces.  Objective  measure¬ 
ments  can  i  rovide  an  understanding  of  the  frequency  and  speed  with  which  par¬ 
ticular  events  occur.  Subjective  judgments  can  provide  information  about  the 
decision-making  process  from  the  participants'  perspective.  And  exj ert  ob¬ 
servations  can.  j-rovide  a  notion  of  decision  soundness  and  accuracy,  when  sue:; 
objective  definitions  are  usually  unavailable. 

What  Is  Being  Compared? 

Before  any  evaluation  can  proceed,  it  is  necessary  to  ask  what  is  bem.: 
compared.  In  a  formal  experiment,  the  comparison  is  between  a  test  and  a 
control  condition.  Similarly,  some  notion  of  a  control  condition  or  at  least 
a  contrasting  condition  is  required  for  aid  evaluation. 

Consider  the  three  interfaces  once  again.  Evaluations  involving  the 
aid/user  interface  are  largely  concerned  with  user  compatibility  and  aid  co¬ 
herence.  Evaluations  could  reasonably  compare  alternate  configurations  ol 
the  aid  or  even  different  aids.  A  comparison  between  an  aid  and  no  aid  is, 
however,  inappropriate.  Questions  about  the  aid/user  interface  assume  a 
decision  aid,  just  as  they  assume  a  user. 

Evaluations  involving  the  user/organization  or  organization/environment 
interfaces  permit  more  comparisons  than  does  the  previous  type  of  evaluation. 

In  particular,  at  these  interfaces  it  is  pos1- "  to  examine  the  organiza¬ 
tion's  operation  both  in  the  presence  and  in  the  absence  of  the  aid.  To 
perform  such  an  evaluation  one  must  recognize,  however,  that  the  definition 
of  the  user — and,  therefore,  the  interface — will  change  when  the  aid  is  not 
present.  In  other  words,  an  aid  is  not  like  a  plug-in  module;  its  introduction 
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will  necessarily  require  changes  in  the  organization.  Thus,  an  effort  to 
compare  performance  when  an  aid  is  present  with  performance  when  the  aid  is 
absent  may  be  confounded  by  effects  of  the  organizational  changes  required 
by  the  aid. 

While  effective  performance  is  likely  to  be  welcome  regardless  of 
whether  it  is  caused  by  the  aid  or  by  the  changes  that  the  aid  requires, 
this  confounding  should  not  be  overlooked.  The  primary  concern  is  that  the 
bulk  of  the  benefit  attributed  to  an  aid  not  be  due  to  its  concomitant  or¬ 
ganizational  changes.  If  it  is  possible  that  the  major  benefit  is  due  to 
organizational  change,  thorough  examination  requires  that  the  evaluator  at¬ 
tempt  to  compare  performance  in  the  modified  organization  in  the  presence 
and  in  the  absence  of  the  aid. 


Summary 

Decision  aid  evaluation  begins  with  a  recognition  of  the  fact  that  the 
aid  will  simply  be  one  component  of  a  more  complex  information-processing 
system.  The  inputs  to  the  aid  and  the  outputs  from  the  aid  will  probably 
travel  through  many  layers  of  a  complex  decision-making  organization.  Thus, 
a  thorough  analysis  of  decision  aid  effects  is  likely  to  examine  aspects  of 
organization  performance  as  well  as  aid  performance. 

In  examining  complex  information-processing  systems  of  this  sort,  it 
is  useful  to  concentrate  on  the  interfaces  between  the  system  components. 
These  are  the  points  at  which  the  system  reveals  itself.  Bottlenecks,  er¬ 
rors,  and  misunderstandings  become  apparent  at  the  interfaces,  and  any 
evaluation  effort  must  strive  to  define  measures  of  effectiveness  that  sense 
or  measure  these  disruptions.  An  effective  aid  is  one  that  increases  the 
speed  with  which  information  can  be  transmitted  across  these  interfaces 
while  decreasing  the  errors  and  misinterpretations  on  the  part  of  the  re¬ 
cipients  of  the  information. 

It  is  useful  to  think  in  terms  of  three  of  the  many  interfaces  within  a 
decision-making  organization: 

•  the  decision  aid  and  user  interface, 

•  the  user  and  decision-making  organization  interface,  and 

•  the  decision-making  organization  and  environment  interface. 

These  interfaces  identify  three  types  of  evaluation  questions: 

•  Is  the  aid  easy  to  use? 

•  Is  the  aid  acceptable  to  the  organization? 

•  Does  the  aid  improve  performance  of  the  organization  in  relation  to 
its  environment? 

These  questions  are  ordered  in  terms  of  their  difficulty  of  evaluation,  with 
aid/user  questions  being  most  amenable  to  experimentation  and  organization,/ 
environment  questions  being  least  open  to  evaluation.  This  ordering  arises 
as  a  result  of  the  level  of  fidelity  required  for  each  type  of  evaluation. 
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Aid/user  evaluations  can  tolerate  low  fidelity,  thereby  decreasing  costs  and 
permitting  a  higher  level  of  experimenter  control.  Organization/environm-nt 
evaluations  require  high  fidelity,  which  both  increases  costs  and  permits 
much  less  experimenter  control.  User/organi zation  evaluations  fall  between 
these  extremes. 

For  each  type  of  evaluation  three  methods  of  collecting  measures  of  ef¬ 
fectiveness  are  available:  objective  measurement,  subjective  judgment,  and 
expert  observation.  Each  method  is  best  suited  to  a  different  notion  of  ef¬ 
fectiveness.  Objective  measures  are  best  suited  to  evaluating  efficiency; 
subjective  judgments  are  best  suited  to  evaluating  likability,  acceptability, 
and  tolerability;  and  expert  observations  are  best  suited  to  evaluating  the 
correctness  of  a  solution  or  inference.  This  association  of  collection  tech¬ 
niques  and  concepts  of  effectiveness  is  simply  a  guideline  and  should  net  be 
interpreted  as  precluding  the  use  of  a  technique  to  evaluate  a  type  of  effec¬ 
tiveness  with  which  it  is  not  associated. 

A  final  question  concerns  what  the  evaluator  intends  to  compare.  In 
some  rare  instances,  it  could  be  appropriate  to  evaluate  an  aid  in  relation 
to  some  absolute  scale  and  remain  unconcerned  about  the  aid's  relation  to 
other  systems,  but  this  is  unlikely.  Just  as  an  experiment  needs  a  baseline 
or  control  condition,  so  will  an  aid  evaluation.  The  comparison  could  be 
between  alternate  designs  of  a  single  aid,  between  opposing  aids,  or  between 
an  aid  and  no  aid,  but  in  most  instances,  some  comp'arison  will  be  required. 
Otherwise,  the  evaluation  results  will  lack  a  context  and,  therefore,  a  basis 
for  deciding  whether  the  aid  is  worthwhile. 

These  various  elements  of  aid  evaluation  (i.e.,  measures  of  effective¬ 
ness,  settings,  and  methods  of  collection)  are  basic  to  the  problem  of  con¬ 
ducting  an  evaluation,  but  they  do  not  represent  all  that  is  involved.  Only 
through  careful  thought  and  effort  can  an  evaluator  pull  these  elements  to¬ 
gether  for  his  or  her  specific  problem.  The  purpose  of  this  section  has 
been  to  point  the  way.  The  hard  work  still  remains. 
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